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1.0 


INTRODUCTION 


To  meet  future  operational  needs •  the  Air  Force  is  currently 
developing  technology  for  a  revolutionary  crew  station  design,  often 
referred  to  as  the  "Super  Cockpit."  Central  to  the  program  is  the 
"virtual  cockpit"  concept  in  which  information  from  many  different 
sources  is  used  to  create  a  "virtual  world"  around  the  pilot.  An 
overall  block  diagram  of  the  system  is  shown  in  figure  1.1.  The  "visual 
world"  will  be  created  by  a  helmet-mounted  virtual  panoramic  display 
(VPp).  Helmet  position  and  orientation  will  be  measured  with  a  magnetic 
position  detection  system  and,  to  function  as  envisioned,  a  helmet- 
mounted  eye  tracker  must  be  included  to  measure  eye  1 ine-of-gaze. 

Eye  tracker  measurements  will  be  used  with  prototype  systems  to 
assist  in  candidate  display  evaluation.  Operationally,  eye  tracking 
will  be  used  for  eye-controlled  switch  selection,  cueing,  eye-slaved 
aiming,  and  pilot  state  monitoring.  Each  of  these  applications  implies 
certain  eye  tracker  performance  requirements. 

Current  eye  tracking  technology  has  been  reviewed  in  Volume  I 
(Review  of  Current  Eye  Movement  Measurement  Technology) .  Eye  tracker 
requirements  and  design  for  the  VPO  application  are  explored  in  this 
volume.  After  reviewing  important  physiological  considerations  and 
defining  eye  tracker  performance  measures,  each  virtual  cockpit  applica¬ 
tion  is  analyzed  in  terms  of  its  eye  tracker  performance  requirements. 

Based  on  the  information  in  the  volume  I  review,  the  analysis  of 
specific  application  requirements,  and  experience  at  Applied  Science 
Laboratories,  a  pupil -to-corneal  reflex  approach  is  proposed  as  the  most 
suitable  technique  for  eye  tracking  on  the  VPD  helmet.  It  is  further 
recotmiended  that  a  two-dimensional  solid  state  array  be  used  for  the  eye 
tracker  detector,  that  an  initial  eye  tracker  design  be  monocular,  and 
that  the  initial  eye  tracker  have  a  60  Hz  update  rate  with  future  enhance¬ 
ments  for  faster  update.  Optical  path,  algorithm,  and  processor  design 
approaches  are  discussed,  and  a  prototype  development  project  is  outlined. 


2.0  TYPES  CF  EYE  MOVEMENT 
2.1  Fixations  and  Saccades 


During  normal  scanning  of  a  visual  scene,  eye  movement  is  characterized 
by  a  series  of  stops  and  very  rapid  jumps  between  stopping  points.  The 
stops,  which  normally  last  at  least  200  msec,  are  called  fixations,  and 
it  is  during  these  fixations  that  most  visual  information  is  acquired 
and  processed.  The  rapid  jumps  between  fixation  points  are  called 
saccades.  Saccades  are  conjugate  eye  movements  (both  eyes  move  together) 
that  can  range  from  1  to  50  degrees  of  visual  angle,  generally  have 
durations  of  30  to  120  msec,  and  achieve  velocities  as  high  as  400-600 
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primarily  because  of  the  very  fast  motion  of  images  across  the  retina, 
and  an  associated  elevated  visual  threshold  just  prior  to  and  during  a 
saccade,  i.e.,  visual  image  supression. 


Figure  l.l  Super  Cockpit  System  Hardware  Block  Diagram  (ref.  1) 


The  eyes  are  not  completely  stationary  during  fixations,  but  exhibit 
a  variety  of  small  Involuntary  motions,  usually  of  less  than  one  degree 
of  visual  angle  magnitude,  called  flicks,  drifts,  and  tremo'^  as 
described  in  section  2.2. 

Smooth  pursuit,  compensatory  eye  movement,  vergence,  and  nystagmus 
are  nonsaccadic  eye  movements  of  relatively  large  magnitude. 

2.2  Smooth  Pursuit 

The  eye  can  smoothly  track  targets  that  are  moving  In  the  range  of  1 
to  30  degrees  per  second.  Some,  but  not  all,  subjects  can  track  targets 
up  to  about  75  degrees  per  second.  These  conjugate,  slow  tracking  eye 
movements  are  usually  called  smooth  pursuit  and  act  to  partially  stabilize 
slowly  moving  targets  on  the  retina.  Slow,  smooth  eye  movements  cannot 
generally  be  executed  voluntarily  without  the  presence  of  a  slowly  moving 
target,  however,  voluntary  control  has  been  demonstrated  after  training. 

2.3  Compensatory  Eye  Movement 

Compensatory  eye  movements  are  conjugate  eye  motions  which  partially 
stabilize  the  visual  field  during  either  active  or  passive  head  or  trunk 
motions.  Such  eye  motions  are  involuntary  and  are  in  response  to  inertial 
information  from  the  vestibular  system  and  head  motion  information 
supplied  by  proprioceptive  sensors  in  the  neck. 

2.4  Vergence 

Vergence  eye  movements  are  nonconjugate  eye  movements  needed  to  keep 
the  visual  scene  in  corresponding  positions  on  both  retinas.  For 
example.  If  a  person  is  fixating  (foveating)  a  target  point  and  the 
target  begins  moving  closer  to  the  person,  the  eyes  must  converge  (both  must 
rotate  toward  their  nasal  side)  to  maintain  the  image  on  corresponding 
parts  of  both  foveas  and  thereby  retain  fusion  of  the  image.  Vergence  eye 
movements  have  a  range  of  about  15  degrees  of  visual  angle  and  maximum 
velocities  of  about  10  degrees  per  second. 

2.5  Nystagmus 

Nystagmus  is  an  involuntary  sawtooth  pattern  of  conjugate  movement 
that  occurs  in  response  to  apparent  motion  of  the  visual  field  (especially 
the  peripheral  field]  or  Inertial  rotation  of  the  body.  Each  cycle  is 
characterized  by  a  "slow  phase"  In  which  the  eyes  move  so  as  to  stabilize 
the  visual  field  on  the  retina,  followed  by  a  return  saccadic  jump  or  fast 
phase.  The  slow  phase  velocity  and  frequency  of  the  pattern  are  related 
to  the  motion  speed  of  the  visual  field  or  the  speed  of  head  rotation  up 
to  maximum  velocities  of  about  100  deg/sec  and  nystagmus  frequency  of 
about  5  Hz.  The  amplitude  is  generally  between  1  and  10  degrees  of  visual 
angle.  Onset  and  changes  in  slow  phase  velocity  of  nystagmus  are  often 
similar  to,  although  not  identical  to,  a  person's  perception  or  sensation 
of  angular  rotation  velocity.  Pathological  nystagmus,  which  can  occur  at 
certain  gaze  angles  or  head  positions,  in  either  eye,  are  important  to 
clinical  diagnosis  in  neurology,  ophthalmology,  and  otolaryngology. 
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2.6  Flicks,  Drifts,  and  Tremor 

At  least  three  types  of  small,  involuntary  eye  motions  commonly 
occur  during  eye  fixations: 

-  Flicks  or  microsaccades  are  very  rapid,  involuntary, 
saccade-like  motions  of  less  than  1  degree  which  may  be 
separated  by  as  little  as  30  msec,  and  tend  to  recenter  images  on 
the  fovea. 

-  Drifts  are  very  small,  slow  (on  the  order  of  0.1  deg/sec), 
apparently  random  motions  of  the  eye. 

-  Tremor  is  a  tiny  (less  than  30  arc  seconds)  high  frequency 
(30-150  Hz)  eye  vibration. 


3.0  PHYSIOLOGICAL  PERFORMANCE  CONSIDERATIONS 

3.1  Pol nt-of-Regard 

For  most  virtual  cockpit  eye  tracker  applications,  the  real  quantity 
of  interest  is  the  pilot's  poi nt-of-regard.  We  want  to  know  what  the 
pilot  is  "looking  at."  The  quantity  actually  measured  by  eye  trackers 
is  the  direction  of  the  eye's  visual  axis,  relative  to  the  environment, 
in  other  words,  eyeball  pointing  direction. 

Fixating  a  point  in  the  visual  field  Implies  positioning  the  eye  so 
that  the  image  of  that  point  falls  on  the  fovea.  The  fovea  is  a  small 
roughly  disk-shaped  area  on  the  retina  th?t  offers  the  highest  visual 
acuity  and  has  a  diameter  corresponding  to  about  2  degrees  of  visual 
angle.  It  is  possible  to  detenriine  point-of-regard  by  measuring  eyeball 
pointing  direction  only  to  the  extent  that  people  always  put  the  target 
they  want  to  "look  at*  on  the  same  spot  within  the  fovea. 

People  can  consistently  position  stationary  targets  on  the  same 
retinal  spot  to  within  1  degree  of  visual  angle.  Miniature  eye 
movements,  however,  imply  at  least  a  minimum  retinal  dead  zone.  Slow 
drift  typically  results  in  2  to  4  arc  minutes  of  eye  motion  during 
fixation,  but  can  approach  10  arc  minutes  or  more  of  motion  during  long 
fixations.  Microsaccades  are  typically  about  5  arc  minutes,  but  can 
sometimes  be  at  least  0.5  degree.  It  has  been  shown  that,  under  proper 
stimulus  conditions,  most  people  can  voluntarily  suppress  microsaccades 
and  maintain  fixation  with  slow  drift  alone  (ref.  2),  but  this  requires 
conscious  effort  and  does  not  represv?nt  normal  scanning  behavior. 

Figure  3.1  shows  examples  of  the  area  covered  by  the  eye's  visual 
axis  during  fixations.  Figures  3.2,  3.3,  and  3.4  show  time  recordings 
of  eye  position  during  fixations.  Notice,  from  figure  3.4,  that  eye 
position  deviation  sometimes  increases  with  fixation  duration.  All  of 
these  measurements  were  made  using  very  accurate  contact  lens  or  corneal 
reflex  tracking  devices  with  the  subject's  head  rigidly  stabilized. 


4 


H 


•V 


Figure  3.1  Area  traversed  by  the  visual  axes  of  two  different  subjects 
during  10  second  fixations;  from  Bennet-Clark  (ref.  3). 
Contours  enclose  the  densest  25X,  50*,  and  75*  of  the  record. 

These  data  represent  relatively  long  fixations.  During  normal 
scanning,  fixations  tend  to  average  about  250  msec,  however,  the  visual 
switch  selection  and  cueing  tasks  associated  with  eye  tracking  In  the 
vlr'  lal  cockpit  may  sometimes  Involve  fixations  of  1  second  or  longer. 

The  data  cited  above  deal  with  motion  within  a  single  fixation.  It 
Is  also  not  clear  that  people  fixate  differently  positioned  targets  with 
exactly  the  same  retinal  region,  nor  Is  It  known  If  people  position 
targets  of  different  shape,  contrast,  luminance,  contextual  content, 
etc.,  at  exactly  the  same  area  on  the  fovea. 


Figure  2.1  Typical  horizontal  eye  position  and  velocity  during  a 
fixation;  from  Elezenman,  Frecker,  and  Hallet  (ref.  4). 

If  a  target  already  Imaged  on  the  fovea  suddenly  moves  by  a  small 
amount,  It  has  been  shown  that  the  probability  of  a  corrective  saccade 
occurring  In  a  given  amount  of  time  decreases  for  decreasing  amounts  of 
target  motion.  Using  such  data.  Young  (ref.  7)  postulated  an  effective 
foveal  dead  zone  within  which  a  target  motion  Is  not  likely  to  produce  a 
corrective  saccade.  He  estimated  this  dead  zone  or  "Indifference  threshold" 
to  be  approximately  +0.3  degree  visual  angle. 

It  has  also  been  shown  by  Bergin  and  Julez  (ref.  8),  Julez  et  al . 
(ref.  9)  and  Scinto  (ref.  JO)  that,  during  a  given  fixation,  focal  attention 
can  be  directed  to  different  areas  within  the  fovea  or,  given  sufficient 
acu  v-y  for  the  target  of  Interest,  even  outside  the  fovea.  Under  normal 
conditions.  If  focal  attention  Is  sustained  on  an  eccentric  target.  It  will 
probably  result  In  a  saccade  to  center  the  target  within  about  150  msec  to  1 
sec,  depending  on  distance  from  the  foveal  dead  zone  (ref.  11). 
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Figure  3.3  Typical  horizontal  and  vertical  eye  motion  during  a 
fixation;  from  Gaarder  (ref.  5). 

We  conclude  that.  In  an  operational  or  training  environment,  It  does  not 
seem  reasonable  to  expect  the  eye  visual  axis  to  Indicate  pol nt-of-regard 
consistently  to  within  better  than  0.3  degree  visual  angle.  Any  eye 
tracking  error  (error  In  determining  the  position  of  the  visual  axis)  must, 
of  course,  be  added  to  this. 

Note  that  it  is  possible  for  people  to  consistently  perform  visual 
aiming  tasks  with  far  better  accuracy  than  0.3  degree.  This  Is  because 
aiming  tasks  usually  do  not  require  positioning  a  target  at  a  certain 
point  on  the  retina,  but  rather  determining  the  relative  alignment  of 
two  or  more  elements  in  the  visual  field. 

3.2  Smooth  Pursuit 

Aiming  and  target  designation  will  be  important  eye  tracker 
applications  In  the  virtual  cockpit  and  will  often  Involve  targets  that 
are  moving  across  the  visual  field.  The  eye  can  smoothly  track  targets 
op  to  about  30  deg/sec,  but  with  somewhat  less  accuracy  than  when 
fixating  stationary  targets. 

People  are  usually  unable  to  precisely  match  target  velocity  at  even 
very  slow  target  speeds  and,  as  target  speed  Increases,  tracking  errors 
tend  to  be  periodically  corrected  with  catch  up  saccades  (ref.  12).  At 
target  speeds  greater  than  30  deg/sec,  smooth  tracking  usually 
degenerates  to  a  series  of  saccades,  although  some  people  can  smoothly 
follow  up  to  75  deg/sec. 

The  pursuit  system  Is  quite  nonlinear  and  smooth  pursuit  dynamics 
vary  considerably,  depending  on  target  dynamics,  target  predictability, 
and  other  factors.  In  general,  the  pursuit  system  has  low  pass 
characteristics,  with  a  corner  frequency  on  the  order  of  1.5  Hz  for 
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Figure  3.4  Eye  position  traces  during  10  second  (a),  30  second  (b), 
and  1  minute  (c)  fixations;  from  Yarbus  (ref.  6). 

predictable  targets  and  1.0  Hz  for  unpredictable  targets.  As  the 
spectral  content  of  the  target  motion  increases,  pursuit  system  lag 
tends  to  increase  (ref.  13  and  14).  There  is  a  latency  of  about  130 
msec  before  the  eye  begins  to  track  a  moving  target.  If  the  target  has 
a  low  luminance  or  low  contrast,  the  latency  will  be  greater.  If  the 
onset  of  target  motion  is  predictable,  this  latency  is  reduced  and,  in 
fact,  there  may  even  be  anticipatory  smooth  pursuit  before  target  motion 
begins,  or  changes  speed  or  direction  (ref.  15  and  16). 

A  typical  smooth  tracking  response  is  shown  in  figure  3.5. 
Unfortunately,  smooth  tracking  data  are  usually  presented  in  the 
literature  in  terms  of  velocity  error  and  frequency  response 
characteristics,  and  not  in  terras  of  retinal  error.  This  information 
cannot  easily  be  used  to  deduce  position  errors.  We  can  estimate, 
however,  that  smooth  pursuit  of  targets  moving  at  over  10  deg/sec  may 
lead  to  retinal  eccentricity  (target  distance  from  foveal  center)  of  1 
degree  or  more  at  least  some  of  the  time.  A  more  accurate  estimate  will 
require  further  research. 
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Figure  3.5  Horizontal  smooth  pursuit  eye  movements  with  eye  position 
traces  displaced  5.7  degrees  above  stimulus  traces  for 
clarity,  from  Hallett  (ref.  12)  after  Lightstone  (ref.  17). 
Initial  velocity  of  ramp  and  sinusoid  stimuli  are  12  and  18 
deg/sec  respectively.  Solid  arrows  show  catch-up  saccades 
and  open  arrows  show  velocity  reversal  points. 

3.3  Range 

The  range  over  which  people  foveate  (the  range  of  eye  motion)  is 
about  +50  degrees  horizontally  and  about  +40  degrees,  -60  degrees 
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vertically  with  respect  to  the  head  (ref.  18).  There  may  be  some 
tendency  to  foveate  less  accurately  near  the  peripheries  of  this  field 
than  in  the  more  central  regions  and  maintainance  of  eccentric  eye 
positions  begins  to  become  uncomfortable  beyond  about  40  degrees.  The 
instantaneous  ambinocular  fiel d-of-view  (area  that  can  be  seen  by  either 
eye)  with  head  and  eyes  fixed  in  a  central  position  is  about  +95  degrees 
horizontally  and  about  +45  degrees,  -65  degrees  vertically.  The 
vertical  field  asymmetry  is  due  to  obstruction  from  the  orbital  arch 
above  the  eye.  The  total  ambinocular  fiel d-of-view  with  respect  to  the 
head  is  about  +165  degrees  horizontally  and  +66,  -82  degrees  vertically 
(ref.  19). 

3.4  Pupil  Diameter 

Pupil  diameter  varies  over  the  population  and  responds  to  a 
multitude  of  physical  and  psychological  stimuli.  Full  pupil  diameter 
range  is  from  about  2  mm  to  10  mm.  The  most  significant  and  predictable 
influence  on  pupil  diameter  is  luminance  of  the  visual  field.  Figure 
3.6  shows  the  typical  relation  between  uniform  visual  field  luminance 
and  pupil  diameter.  When  steady  state  luminance  is  over  1000  ml 
(daylight  conditions),  pupil  diameter  values  between  2  and  3  mm  can 
usually  be  expected.  Retinal  adaptation  to  ambient  illumination  tends 
to  return  the  pupil  diameter  toward  its  middle  range. 


Figure  3.6  Pupil  diameter  as  a  function  of  uniform  visual  field 
luminance  (from  Roth  and  Finkel stein,  ref.  19). 
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Pupil  diameter  does  respond  to  other  factors,  including  fatigue, 
stress,  state  of  arousal,  etc.  It  has  sometimes  been  casually  observed 
by  the  author  that  the  pupil  tends  to  constrict  during  eye  tracker 
calibration  procedures  when  subjects  must  concentrate  on  fixating 
certain  target  points  and  tends  to  dilate  when  a  subject  actually  begins 
to  fly  an  aircraft  simulator  and  is  apparently  stimulated  by  the 
"exciting"  nature  of  the  task. 


4.0  EYE  TRACKER  PERFORMANCE  MEASURES 

4.1  General  Concepts 

Terms  specifying  performance  are  sometimes  used  differently  by 
different  people  and  in  different  fields.  The  following  discussions 
define  these  terms  for  the  purpose  of  eye  tracker  specification. 

4.1.1  Accuracy  and  Precision 

As  a  general  concept,  accuracy  is  the  difference  between  true  line- 
of-gaze  and  measured  1 ine-of-gaze,  usually  expressed  in  terms  of  visual 
angle.  It  is  a  measure  of  how  good  the  absolute  eye  position  measure 
is,  and  there  are  many  different  specific  ways  in  which  this  concept  may 
be  defined.  Precision  can  be  thought  of  as  the  amount  of  instrument 
noise  (jitter)  in  the  eye  position  measure  when  the  observer  attempts  to 
keep  the  eye  perfectly  stationary. 

As  discussed  in  section  3,  a  real  eye  ’s  never  perfectly  stationary 
and  when  people  fixate  a  spot,  they  do  not  center  it  on  the  fovea  with 
zero  error.  If  a  real  eye  is  used  to  measure  eye  tracker  accuracy  and 
precision,  this  physiological  uncertainty  must  be  taken  into  account. 

It  is  also  important  to  be  wary  of  tests  using  model  (artificial) 
eyes.  It  is  extraordinarily  difficult  to  duplicate  all  the  properties 
of  a  real  eye  and  performance  achieved  with  a  model  eye  may  be  misleading. 

Specific  accuracy  and  precision  definitions  for  the  virtual  cockpit 
application  are  proposed  in  section  4.2. 

4.1.2  Linearity 

Linearity  is  the  extent  to  which  a  change  in  device  output  is 
proportional  to  change  in  eye  position.  Stated  another  way,  it  is  the 
degree  to  which  a  plot  of  device  output  versus  true  eye  position  tends 
to  be  a  straight  line.  The  deviation  from  a  straight  line  is  generally 
specified  as  a  percent  of  the  size  of  the  excursion. 

Note  that  a  device  can  have  a  large  offset  error  and  still  be  linear 
as  long  as  changes  in  output  retain  proportionality  to  changes  in  eye 
position.  When  an  application  requires  that  a  change  or  rate  of  change 
be  determined,  as  opposed  to  absolute  position,  linearity  is  sometimes 
specified  instead  of  accuracy.  For  the  virtual  cockpit  application, 
accuracy  is  a  more  appropriate  specification. 
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4.1.3  Resolution 


Resolution  specifies  how  finely  the  output  scale  is  divided  or,  in 
other  words,  the  smallest  change  that  can  ever  be  observed  in  the 
measured  value.  Resolution  is  introduced  by  digitization  of  a  process 
and  has  no  meaning  for  analog  processes. 

4.1.4  Update  Rate  and  Transport  Delay 

Measurement  update  rate  specifies  the  number  of  independent  data 
samples  per  second  provided  by  the  instrument.  The  Nyquist  criterion 
tells  us  that,  in  order  to  allow  complete  recovery  of  an  input  signal, 
data  sampling  frequency  must  be  at  least  twice  the  highest  significant 
input  frequency  component.  This  criterion  assimies  that  an  ideal  filter, 
which  is  actually  net  realizable,  operates  on  the  sampled  output.  In 
most  real  situations,  sample  rate  should  be  significantly  more  than 
twice  the  highest  important  frequency  component. 

Transport  delay  specifies  the  temporal  relationship  of  each  data 
sample  to  the  actual  event  being  measured.  For  example,  a  system  might 
provide  1000  data  samples  per  second,  but  the  entire  data  stream  might 
be  delayed  by  1  second.  Thus,  an  event  at  t  =  0  sec  would  appear  in  the 
data  sample  output  at  t  *  1  sec  and  an  event  at  t  =  0.001  sec  would 
appear  in  the  data  sample  output  at  t  =  1.001  second,  etc.  When  an  eye 
tracking  device  is  referred  to  as  a  "60  Hz"  system,  for  example,  it 
usually  means  that  the  update  rate  is  60  samples  per  second  and  usually 
implies  nothing  about  transport  delay. 

For  applications  that  use  the  data  to  control  or  switch  something  in 
real  time,  the  amount  of  transport  delay  is  very  important.  On  the 
other  hand,  if  data  is  to  be  used  only  off-line  (studies  after  the  fact 
for  research,  etc.)  transport  lag  does  not  matter  at  all,  since  it  can 
simply  be  subtracted  out. 

Transport  delay  does  not  affect  the  frequency  content  or  "wave  form" 
of  the  information,  it  just  determines  when  the  information  is  received. 
Update  rate  does  affect  the  information  content.  A  slow  update  rate 
will  cause  high  frequency  information  to  be  irretrievably  lost.  It  can 
also  interfere  with  the  recovery  of  lower  frequency  information  by 
"aliasing"  from  the  higher  frequency  signals. 

4.2  Specific  Accuracy  and  Precision  Definition 

As  previously  noted,  accuracy  can  be  specified  in  many  ways.  The 
definition  used  should  be  compatible  with  real  eye  movement  behavior  and 
with  practical  means  for  testing.  If  testing  is  done  with  a  real  eye, 
we  can  be  most  confident  of  knowing  true  point-of-gaze  when  a  person  is 
asked  to  fixate  a  small  stationary  target  for  a  short  time.  The  most 
straightforward  procedure  for  gathering  such  data  is  to  have  subjects 
fixate  a  set  of  predefined  target  points  while  eye  tracker  data  are  being 
recorded  and  then  to  compute  the  mean  and  standard  deviation  of  the 
measurement  during  each  fixation. 


From  figure  4,1  we  can  define  fixation  measurement  error  as  the 
difference  between  true  eye  position  and  mean  computed  eye  position 
during  a  fixation.  An  operational  accuracy  definition  often  used  at 
Applied  Science  Laboratories  is  the  expected  (average)  fixation 
measurement  error.  A  more  rigorous  definition  would  specify  a  maximum 
fixation  measurement  error  rather  than  the  expected  value.  This  can  be 
expressed  statistically  as  a  95%  confidence  interval. 

Precision  can  be  defined  as  a  confidence  interval  about  the  mean 
during  fixations.  Again,  assuming  a  normal  distribution,  precision  can 
be  expressed  in  terms  of  the  standard  deviation  of  the  measurement  (a,,^) 
during  each  fixation  (see  figure  4.1). 

PROBABILITY 
DENSITY  OP 
MEASURED 
EYE  POSITION 


FIXATION  MEASUREMENT 
ERROR 

Figure  4.1  Fixation  measurement  error  and  standard  deviation  during  a 
single  fixation. 

For  the  purpose  of  this  report,  we  will  use  the  following 
operational  definitions: 

-  Fixation  Measurement  Error  --  difference  between  true  eye  line- 
of-gaze  and  mean  computed  line-of-gaze  during  a  fixation  of  some 
specified  duration. 


-  Accuracy  --  95%  confidence  interval  for  fixation  measurement 
error.  Assuming  a  normal  distribution,  this  can  be  estimated  as 
^°FME’  ®FME  standard  deviation  of  fixation 

measurement  error. 


VISUAL 

ANGLE 
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-  Precision  --  95%  confidence  interval  about  the  mean  measured 

value  during  each  fixation.  Assuming  a  normal  distribution,  this 
can  be  estimated  by  2o|v|  where  of/j  is  the  standard  deviation  of  the 
eye  position  measurement  during  fixations. 

If  tests  are  performed  with  real  eyes,  it  must  be  understood  that 
such  tests  will  not  strictly  determine  eye  tracker  performance,  but 
rather  the  total  accuracy  and  precision  with  which  the  eye  tracker  can 
measure  a  person's  poi nt-of-regard.  The  physiological  variability 
discussed  in  section  3  will  be  included.  Of  course,  total  performance 
is  ultimately  the  quantity  of  Interest. 

A  value  between  0.5  and  1.0  second  is  probably  a  reasonable  choice 
for  the  fixation  duration  used  to  compute  accuracy  and  precision  (see 
fixation  measurement  error  definition  above).  This  range  is  consistent 
with  maximum  fixation  durations  that  may  occur  during  eye  controlled 
switching  and  aiming  tasks,  but  is  not  so  long  that  it  is  difficult  to 
maintain  fixations. 

If  data  are  examined  over  an  entire  fixation,  we  can  be  confident  of 
knowing  point-of-regard  (location  of  the  target)  to  within  the  accuracy 
limit.  The  probability  that  a  single  measurement  sample  will  be  within 
a  certain  distance  of  true  point-of-regard  depends  on  both  accuracy  and 
precision.  Assuming  normal  distributions  and  independence  of  the  two 
components,  the  uncertainty  associated  with  accuracy  and  precision  can 
be  combined  by  adding  variances  ^[:mc  +0*^},^).  A  95%  confidence 
interval  (Cl),  or  95%  circular  probaoTe  error,  for  a  single  data  sample 
can  be  computed  as 

Cl  =  IaCC^  +  PREC^  (1) 

where  ACC  and  PREC  are  accuracy  and  precision  as  defined  above. 

For  the  virtual  cockpit  application  we  will  be  dealing  with  a 
helmet-mounted  eye  tracker  measuring  eye  position  with  respect  to  the 
helmet,  a  helmet  mounted  display,  and  a  helmet  position  and  orientation 
measurement  device. 

When  considering  line-of-gaze  with  respect  to  visual  images  that  are 
defined  with  respect  to  the  helmet,  virtual  images  of  switches  for 
example,  eye  tracker  accuracy  and  precision  alone  determine  performance. 

In  cases  where  a  pilot  is  looking  at  an  external  target,  helmet 
position  measurement  errors  must  also  be  taken  into  account.  For  a 
single  line-of-gaze  measurement  with  respect  to  the  cockpit,  the  95% 
confidence  interval  (CT)  is 

Cr  =  +  PREC^  +  Cl^hd  (2) 

where  is  a  95%  confidence  interval  (or  circular  probable  error)  for 
the  helmet  orientation  measurement. 
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In  the  special  case  of  simulator  applications  for  which  all  Imagery 
is  presented  on  the  helmet  mounted  display,  we  never  have  to  consider 
helmet  orientation  measurement  errors  in  connection  with  eye  tracking. 

The  display  system  will  always  "know"  where  an  image  is  with  respect  to 
the  helmet.  A  helmet  orientation  measurement  error  might  cause  the 
image  to  be  placed  in  the  wrong  spot  on  the  helmet  display  and  will 
affect  the  fidelity  of  simulation,  nevertheless,  poi nt-of-regard  errors 
need  to  be  considered  only  with  respect  to  the  actual  position  of  the 
displayed  image. 

4.3  Performance  Enhancement  Techniques 

Because  of  the  physiological  variability  discussed  in  section  3  and 
because  of  measurement  limitations,  it  may  not  always  be  possible  to 
make  an  open  loop  point-of-regard  measurement  with  desired  accuracy  and 
precision.  It  may  sometimes  be  possible  to  close  the  control  loop  and 
enhance  performance,  either  by  showing  pilots  their  own  poi nt-of-gaze 
(section  4.3.1),  or  by  giving  the  pilots  a  fine  control  adjustment  that 
is  independent  of  eye  motion  (section  4.3.2). 

During  smooth  pursuit  or  compensatory  eye  movements,  position  errors 
in  point-of-regard  measurement  will  be  introduced  with  lag  or  transport 
delay  in  the  eye  tracker  or  other  parts  of  the  visual  display  system. 
These  errors  can  be  reduced  by  careful  lead  compensation  (section 
4.3.3). 

4.3.1  Secondary  Visual  Feedback 

Secondary  Visual  Feedback  (SVFB)  refers  to  a  display  showing  a 
person  his  own  measured  poi nt-of-gaze.  In  the  case  of  the  virtual 
cockpit,  this  would  consist  of  a  cursor,  circle,  or  highlighted  area 
displayed  by  the  Virtual  Panoramic  Display  at  the  point  calculated  by 
the  eye  tracker  to  be  the  point-of-regard. 

It  was  shown  by  Peli  and  Zeevi  (ref.  20)  that  SVFB  can  improve 
smooth  pursuit  of  targets  moving  at  1-10  deg/sec  with  respect  to  the 
visual  field.  It  has  also  been  shown  by  Kenyon  et  al  .  (ref.  21)  that 
smooth  eye  movements  can  be  generated  by  most  people  in  the  absence  of  a 
target,  and  that,  when  a  target  temporarily  disappears,  people  are 
better  able  to  maintain  smooth  pursuit  tracking  when  SVFB  is  provided. 
These  data  imply  a  potential  improvement  in  conscious  fine  control  of 
eye  movement  with  SVFB.  The  research  indicates,  however,  that  the  above 
effects  can  degenerate  when  delay  in  the  SVFB  display  is  greater  than  50 
or  60  msec  and  when  retinal  error  of  the  SVFB  becomes  larger  than  the 
fovea.  Use  of  SVFB  does  seem  to  require  concentration,  and  most 
experiments  have  included  some  practice  time  for  subjects.  There  is  a 
natural  tendency  to  "chase"  an  SVFB  display  that  is  a  little  off-center 
with  respect  to  the  fovea.  This  effect  may  either  lead  to  undesired  eye 
movements  away  from  the  targets  or  an  undesirable  extra  mental  effort  to 
suppress  such  movements. 

SVFB  may  prove  to  be  very  beneficial  in  conjunction  with  the  VPD/eye 
tracker  by  enabling  pilots  to  correct  for  small  eye  tracking  errors  and 


Improve  the  effective  system  accuracy.  On  the  other  hand,  the  somewhat 
unnatural  task  of  attempting  to  superimpose  an  SVFB  cursor  on  a  visual 
target  may  prove  frustrating  and  annoying.  Research  will  be  needed  to 
determine  the  efficacy  of  SVFB  for  performance  enhancement  of  eye 
tracking  in  the  virtual  cockpit. 

Even  if  SVFB  proves  not  to  be  effective  for  enhancing  performance, 
pol nt-of-gaze  feedback,  in  the  form  of  a  cursor,  circle,  or  highlighted 
area,  should  still  be  available  to  enable  the  pilot  to  monitor  eye 
tracker  performance,  when  desired. 

4.3.2  Head-Slaved  Fine  Positioning 

Conscious  fine  control  of  eye  position  1s  not  a  task  that  comes 
naturally  to  people,  even  when  secondary  visual  feedback  is  provided. 
Fine  conscious  control  of  head  position  is  a  far  more  natural  task,  as 
is  alignment  of  two  visual  images. 

If  a  cursor  or  aiming  reticle  that  is  normally  slaved  to  eye  motion 
(SVFB  display)  is  frozen  with  respect  to  the  helmet  whenever  the  pilot 
activates  a  switch,  then  the  pilot  can  visually  fixate  a  target  to 
quickly  bring  the  reticle  very  close  to  the  target,  press  the  "freeze" 
button,  and  more  precisely  align  the  reticle  and  target  with  a  small 
amount  of  head  motion.  Although  this  type  of  aiming  would  take  longer 
than  eye  fixation  alone,  it  would  still  be  far  quicker  than  moving  a 
permanently  head-fixed  reticle  all  the  way  to  an  eccentric  target  (head 
slaved  aiming)  and  may  allow  reduction  of  eye  and  head  tracking  errors. 
The  small  head  motions  involved  would  probably  be  possible  even  under 
high  G's.  It  is  not  being  claimed  that  the  head  can  be  pointed  more 
precisely  than  the  eyes,  but  rather  that  errors  in  measuring  head  and 
eye  position  might  be  more  easily  reduced  with  head  controlled  visual 
feedback  than  with  eye  controlled  feedback. 

The  head  slaving  feature  can  be  effective  only  when  a  visual  target 
is  relatively  stable  with  respect  to  the  visual  field,  since  head 
tracking  has  been  shown  to  be  effective  only  at  frequencies  below  about 
0.8  Hz  (ref.  22  &  23). 

4.3.3  Lead  Compensation 

Delays  or  lags  in  the  eye  tracker  and  display  system  will  result  in 
position  errors  during  eye  movement.  For  example,  if  the  eye  is 
rotating  at  constant  velocity,  transport  lag  in  obtaining  and  using  an 
eye  position  measurement  will  produce  a  steady  state  pol nt-of-regard 
measurement  error  equal  to  the  delay  multiplied  by  the  target  speed.  We 
need  to  be  concerned  about  such  errors  only  during  smooth  pursuit  or 
compensatory  eye  motions  when  the  pilot  will  presumably  be  attempting  to 
maintain  poi nt-of-regard  on  a  visual  target. 

The  delays  in  the  system  will  be  known  and,  therefore,  the  delay 
error  can  be  reduced  with  a  lead  compensation  scheme.  The  potential 
problem  with  lead  compensation  is  that  any  high  frequency  noise  (jitter) 
in  the  measurement  will  be  magnified.  To  avoid  this  problem,  the  signal 
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should  be  low  pass-filtered  with  a  large  enough  time  constant  to  reduce 
the  velocity  precision  error  to  2  or  3  deg/sec.  A  threshold  should  also 
be  applied  so  that  the  lead  term  is  used  only  when  the  smoothed  velocity 
signal  exceeds  2  or  3  deg/sec. 

In  the  constant  eye  velocity  example  used  above,  there  will  be  a  lag 
in  reducing  the  delay  error  that  Is  approximately  equal  to  the  filter 
time  constant  and  a  similar  overshoot  when  the  eye  motion  stops.  Since 
the  velocity  filtering  introduces  a  lag  term,  this  type  of  compensation 
makes  sense  only  If  the  smooth  pursuit  and  compensatory  eye  motions  of 
Interest  have  a  frequency  spectrum  with  a  signficantly  lower  bandwidth 
than  that  of  the  lag  term.  As  discussed  In  section  3,  the  smooth 
pursuit  system  usually  exhibits  a  corner  frequency  on  the  order  of  1.0 
Hz,  and,  in  the  virtual  cockpit  applications,  we  will  probobly  be 
dealing  with  even  lower  bandwidths.  A  velocity  filter  time  constant  of 
about  50  msec  (3  Hz  corner  frequency)  should  be  appropriate,  and  will 
probably  be  sufficient  to  suitably  reduce  velocity  measurement  noise. 

To  avoid  producing  overshoot  errors  after  saccadic  motions,  the 
logic  can  be  disabled  whenever  eye  velocity  is  greater  than  50  or  60 
deg/sec. 


5.0  VIRTUAL  COCKPIT  EYE  TRACKING  APPLICATIONS 

The  virtual  panoramic  display  (VPD)  is  the  visual  portion  of  the 
virtual  cockpit.  The  display  optics  will  be  helmet-mounted  and  the 
helmet  will  incorporate  a  magnetic  position  detection  system  and  an  eye 
tracker.  The  specific  tasks  requiring  eye  tracker  data  will  be 
evaluation  of  candidate  displays,  cueing,  eye-controlled  switching,  eye- 
slaved  aiming,  and  pilot  state  monitoring.  The  following  subsections 
discuss  general  eye-related  tracker  requirements  for  the  virtual  cockpit 
application,  the  specific  eye  tracker  performance  requirements  for  each 
task,  and  the  resulting  combined  performance  specification. 

5.1  General  Considerations 

The  virtual  panoramic  display  (VPO)  in  the  virtual  cockpit  will  have 
a  maximum  binocular  field  of  +60  degrees  horizontally  by  +30  degrees 
vertically.  The  two  VPD  prototypes  specifically  considered  in  section  6 
for  eye  tracker  interface,  the  off-aperture  and  dual  mirror  systems, 
have  horizontal  fields  of  +35  degrees  and  +45  degrees  respectively  and 
vertical  fields  of  +18.75  degrees  and  +22.^  degrees  respectively  (ref. 
24).  The  horizontaT  range  covered  by  the  eye  visual  axis  is  about  +50 
degrees,  and  only  about  +40  degrees  with  comfort.  The  vertical  eye” 
movement  range  covers  the  entire  vertical  VPD  display  (see  section  3.3). 
Ideally,  eye  tracker  range  should  encompass  at  least  +40  degrees 
horizontally  by  +30  degrees  vertically.  This  ideal  range  will  probably 
not  be  achievable,  and  will  certainly  not  be  achievable  with  maximum  eye 
tracker  performance.  Maximum  performance  will  probably  be  restricted  to 
a  15  Or  20  degree  radius  inner  field  (see  volume  1,  section  4.5.1). 


The  projected  luminance  of  the  VPO  is  2000-3000  foot-Lamberts 
(ImL  =  .929ft-l).  A  unifonii  field  luminance  in  this  range  would 
probably  result  in  pilot  pupil  diameters  of  2-3  millimeters.  In 
general,  however,  tne  field  will  not,  be  uniform  and  the  projected 
luminance  values  may  not  always  be  achieved.  In  fact,  a  very  brief  test 
with  three  of  the  breadboard  VPO  helmets  (described  later  in  this 
report;  section  6.2.1)  resul ten  in  significantly  larger  pupil  diameters 
under  most  conditions.  When  used  in  real  flight  daylight  conditions, 
the  levels  can  easily  equal  or  exceed  2000  foot-Lamberts,  although  this 
may  be  somewhat  attenuated  by  the  canopy  and  VPO  combiner  optics. 

Minimum  scene  luminance,  at  night  or  in  a  simulator  when  not  much 
information  is  being  displayed  on  the  VPD,  may  be  under  10  foot- 
Lamberts.  Early  on  in  an  eye  tracker  development  project,  pupil 
diameter  should  be  measured  for  a  number  of  subjects  under  the  expected 
range  of  realistic  lighting  conditions. 

Different  pilots  will  have  slightly  different  eye  locations  with 
respect  to  a  given  helmet,  even  though  the  helmet  may  have  a  customized 
bladder  or  other  custom  fitting  technique.  When  a  pilot  first  dons  the 
virtual  panoramic  display  helmet,  the  eye  tracker  optics  will  have  to  be 
adjusted  for  that  pilot's  eye  placement  so  that  a  properly  centered  and 
focused  eye  image  is  obtained.  The  adjustments  wi^l  probably  consist  of 
turning  3  knobs  on  the  helmet  while  viewing  the  eye  camera  image. 

For  simulator  research  applications,  it  is  acceptable  for  these 
adjustments  to  be  made  by  a  technician,  but,  for  later  training  and 
operational  use,  it  should  be  possible  for  the  pilot  to  do  this.  In 
either  case,  the  adjustments  should  be  straightforward  and  require  no 
more  than  1  minute.  There  should  also  be  appropriate  feedba  k  to  assure 
the  simulator  operator  or  pilot  that  proper  performance  is  being 
achieved  by  the  eye  tracker.  When  the  adjustments  are  made  by  the 
pilot,  the  eye  image  with  recognition  indicators  can  be  displayed  on  the 
vPD  during  the  process. 

No  matter  how  good  the  VPD  helmet  fitting  system,  there  will  be  some 
motion  of  the  helmet  with  respect  to  the  head.  Because  the  skin  can 
move  with  respect  to  all  parts  of  the  face  and  head,  slippage  can  be 
totally  eliminated  only  with  tne  use  of  a  bite  bar,  which  is  obviously 
not  acceptable.  It  is,  however,  important  that  the  helmet  fit  as 
snuggly  as  possible  without  causing  headache  or  other  discomfort.  Based 
on  experience  at  ASL  in  other  applications,  it  is  reasonable  to  expect 
that  slippage  of  the  eye  (the  motion  of  an  imaginary  pointer  located  at 
the  eye  and  rigidly  connected  to  the  helmet)  will  be  restricted  to  about 
4  mm.  Kociar,  (ref.  24)  reports  that  well-fitted  flight  helmats  can  be 
expected  to  slip  no  more  than  3  mm.  The  eye  tracker  must  therefore 
accommodate  at  least  3  to  4  mm  of  slippage  to  continue  to  perform 
acceptably. 

In  order  to  achieve  maximum  eye  tracker  accuracy,  it  will  certainly 
be  necessary  fnn  each  pilot  to  go  through  a  calibration  procedure  during 
which  a  number  of  oisplayed  target  points  are  fixated.  It  is  important 
that  it  be  possible  for  the  pilot  to  self-administer  this  procedure,  and 
that  it  take  no  longer  than  1  minute.  There  must  be  feedback  after  the 
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calibration  to  assure  the  pilot  or  simulator  operator  that  the 
calibration  has  worked  properly.  This  feedback  should,  preferably,  be 
in  the  form  of  a  cursor  displayed  to  the  operator  and/or  the  pilot, 
showing  the  pilot's  poi nt-of-gaze  with  respect  to  the  scene.  This  type 
of  feedback  provides  the  clearest  and  quickest  visual  indication  of  how 
good  performance  is  and  the  nature  of  any  problems.  Although  such 
feedback  need  not  necessarily  be  displayed  all  the  time,  it  should  be 
available  at  any  time  after  calibration. 

Nn  matter  how  tight  the  performance  specifications  that  are 
achieved,  it  is  still  prudent  to  allow  for  quick  offset  adjustments  to 
be  made  at  virtually  any  time,  either  by  requiring  the  pilot  to  fixate  a 
displayed  target  point  and  activate  a  switch,  or  by  displaying  measured 
point-of-gaze  to  the  pilot  and  allowing  continuous  adjustments  to  be 
made  with  a  four-way  switch  (left,  right,  up,  and  down). 

A  system  to  be  used  in  flight,  as  opposed  to  just  simulator  use, 
must  also  have  a  small  ruggedized  electronics  and  processor  package, 
preferably  made  with  military  specification  components.  Optical 
components  must  be  mechanically  capable  of  safely  withstanding  whatever 
vibration  and  g  force  will  be  encountered  in  the  aircraft.  The  range  of 
different  scene  luminance  conditions  encountered  in  flight  will  probably 
be  far  greater  than  for  simulator  use.  To  compensate  for  dramatic 
changes  in  ambient  conditions,  automatic  adjustment  of  the  eye  tracker 
illuminator  intensity  might  be  required,  in  addition  to  automatic  sensor 
gain  control  techniques, 

5.2  Evaluate  Candidate  Display  Formats 

Display  evaluation  is  a  research  application  requiring  off-line  data 
analysis.  The  analysis  will  probably  Involve  determining  the  amount  of 
visual  "traffic"  between  different  display  elements,  the  amount  of  dwell 
time  spent  on  various  elements  in  the  display,  the  entropy  (orderliness) 
of  scan  patterns,  and,  perhaps,  the  time  required  for  visual  reaction  to 
certain  displayed  events. 

All  of  these  analysis  tasks  primarily  require  determination  of 
fixation  positions,  start  times,  and  durations.  It  will  be  Important  to 
differentiate  among  fixations  on  adjacent  display  elements.  A  95% 
confidence  Interval  for  a  single  eye  position  measurement  sample  is 
given  by  equation  (1)  in  section  4.2.  If  the  adjacent  edges  of  two 
potential  fixation  targets  are  separated  by  visual  angle  a,  and  we  assume 
that  poi~t-of-regard  is  on  one  of  these  targets  and  not  the  space 
between  triem,  then  a  single  data  sample  from  the  eye  tracker  can  be 
assigned  to  one  or  the  other  with  95%  certainty  if  a/2  is  greater  than 
or  equal  to  Cl  from  equation  (1). 

a/2  > JaCC^  +  PREC^  (3) 

In  the  Cdse  of  off-line  analysis,  we  can  identify  fixations  and 
average  them  over  the  entire  fixation,  thus  eliminating  the  "jitter" 
during  fixations.  Fixations  are  often  defined  as  periods  of  at  least 
100  msec  during  which  the  eye  does  not  move  more  than  about  1  degree 
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visual  angle.  Reduction  of  the  data  to  fixation  points  Is  generally 
part  of  the  off-line  analysis  procedure  anyway,  and  averaging  over  the 
entire  fixation  effectively  eliminates  precision  as  a  consideration. 

All  of  the  remaining  error  Is  encompassed  in  the  accuracy  definition, 
and  we  can  just  require  that: 

0/2  >  ACC  (4) 

The  details  of  display  layout  have  not  yet  been  determined,  but  it 
Is  anticipated  that  major  symbols  will  be  separated  by  about  I. 5  degrees 
visual  angle.  This  figure  would  Imply  an  eye  tracking  accuracy 
requirement  of  about  0.75  degrees  for  the  display  evaluation  task. 

Fixations  are  never  much  shorter  in  duration  than  100  msec  and  are 
usually  longer  than  200  msec.  The  minimum  time  required  to  Initiate  a 
saccade  in  response  to  a  visual  stimulus  has  been  shown  to  be  about  100 
msec.  There  will  probably  be  no  need  to  look  at  the  details  of  saccadic 
velocity  for  candidate  display  evaluation.  A  60  Hz  update  rate  is, 
therefore,  quite  sufficient  for  most  off-line  scan  path  analyses.  In 
fact,  much  research  of  this  type  is  done  with  30  Hz  samples.  Transport 
delay  can  be  subtracted  out  during  off-line  analysis  and  is  not 
relevant. 

5.3  Switching 

Eye-controlled  switch  selection  Is  a  real-time  eye  tracker 
application  that  will  enable  the  pilot  to  toggle  a  switch  essentially  by 
looking  at  It.  The  potential  advantages  over  manual  switching  will  be 
decreased  time  required  to  activate  a  switch,  and  improved  ability  to 
manipulate  switches  under  high  g  conditions  or  when  the  hands  are  busy 
with  other  tasks.  The  protocol  will  probably  require  the  pilot  to  note, 
via  some  sort  of  feedback,  that  the  system  has  computed  point-of-regard 
to  be  on  the  proper  switch,  and  will  then  require  the  pilot  to  confirm 
the  choice  with  a  voice  command  or  a  manual  consent  switch.  With  the 
VPO,  feedback  could  be  some  form  of  highlighting,  a  circle  drawn  around 
the  switch,  or  some  other  visual  indicator.  Additional  feedback  in  the 
form  of  a  cursor  snowing  computed  point-of-gaze  may  or  may  not  prove 
useful . 

The  feasibility  of  this  sort  of  scheme  was  demonstrated  by  Calhoun, 
Janson,  and  Arbak  (ref.  25).  Subjects  performing  a  tracking  task  were 
required  to  activate  one  of  seven  switches  as  soon  as  possible  after 
receiving  an  audio  cue.  For  a  given  set  of  trials,  switch  activation 
was  accomplished  either  manually  (by  pressing  the  appropriate  switch 
with  the  vree  hand),  by  eye-slaved  switching  with  a  manual  consent 
switch,  cr  by  eye-slaved  switching  with  a  verbal  consent  response.  When 
eye-slaved  switching  was  used,  visual  feedback  was  provided  by 
Illuminating  the  chosen  switch.  The  manual  consent  switch  was  a  thumb 
activated  push-button  on  the  tracking  task  control  stick. 

After  the  appropriate  learning  period,  average  switching  time,  from 
the  audio  cue  to  final  switch  activation,  was  essentially  the  same  for 
both  manual  switching  and  eye-controlled  switching  with  manual  consent 
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(1.7  -  1.8  seconds).  Eye-controlled  switching  with  voice  consent  took 
significantly  longer,  due  primarily  to  delays  in  the  voice  recognition 
system  used.  This  result  suggests  that  eye-controlled  switching  may  not 
provide  a  speed  advantage  over  manual  switching,  at  least  not  under  the 
particular  task  loading  conditions  tested.  A  previous  study  by  Calhoun, 
Arbak,  and  Boff  (ref.  26)  shows  similar  switching  times  (1.5  -  1.7 
seconds)  for  a  similar  eye-controlled  switching  task  with  manual 
consent,  and  reported  intermediate  times  measured  from  the  audio  cue,  as 
shown  in  figure  5.1. 


TRACKING  SWITCH  CLOSED 

DISPLAY 


Figure  5.1  Eye-controlled  switching  response  times  for  two  subjects, 
from  Calhoun  et  al .  (ref  26). 

Assuming  the  switches  are  visual  targets  with  boundaries  separated 
by  a  visual  angle  of  at  least  «,  then,  as  explained  in  the  previous 
section,  the  most  conservative  performance  requirement  is 

a/2  lJaC?  +  PREC^  (5) 

The  visual  angle  a/2  can  also  be  interpreted  as  the  minimum  envelope 
around  the  image  of  the  switch  within  which  eye  position  measurements 
will  be  considered  to  be  "on  the  switch."  In  the  Calhoun  et  al . 
experiment  (ref.  25)  this  envelope  was  about  2  degrees  from  the  center 
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of  the  switch.*  If  the  system  averages  computed  eye  position  over  a 
running  time  window  or  passes  the  data  through  a  low  pass  filter  before 
comparison  with  switch  positions,  then  the  precision  component  can  be 
reduced  at  the  expense  of  increased  lag.  If  we  average  over  the  same 
interval  used  to  define  accuracy  and  precision  then,  as  explained  in  the 
previous  section,  we  can  effectively  eliminate  the  precision  error. 

If  eye  position  data  samples  were  completely  independent  of  one 
another  (e.g.,  white  noise)  then,  even  if  we  averaged  over  a  much 
shorter  interval,  the  precision  component  could  be  divided  by  “sTn, 
where  n  is  the  number  of  samples  being  averaged. 

The  component  of  precision  due  to  measurement  noise  will  probably 
look  "white"  over  the  bandwidth  of  the  eye  tracker,  but  the 
physiological  component,  slow  drift  for  example,  will  not.  For 
illustrative  purposes,  we  can  make  a  conservative  guess  and  assign  0.2 
degrees  as  the  precision  component,  (PREC)  not  due  to  relatively  high 
frequency  noise.  If  we  can  average  only  over  a  time  interval 
signficantly  shorter  than  the  fixation  duration  chosen  in  section  4.2, 
our  criterion  then  becomes  _ _ 

a/2  +  (PREC^  -  PREc'^)/n  +  PREc'^  (6) 

where  n  is  the  number  of  samples  averaged.  As  mentioned  in  the  previous 
section,  o  is  likely  to  be  about  1.5  degrees.  Note  that,  by  increasing 
the  sample  rate,  we  can  increase  n  in  equation  (6)  without  increasing 
the  time  interval  over  which  we  average.  The  potential  advantage  is 
that  a  can  be  reduced,  or  accuracy  and  precision  requirements  relaxed. 

It  is  reasonable  to  require  the  eye  position  measurement  to  be 
within  the  switch  envelope  for  some  minimum  time  before  the  switch  is 
activated,  otherwise  the  display  system  would  have  to  respond  whenever  a 
saccade  passed  through  a  switch  area.  On  the  other  hand,  we  ideally 
want  no  more  than  a  barely-perceptible  delay  between  the  beginning  of  a 
fixation  on  the  switch  and  feedback  to  the  pilot,  such  as  highlighting 
of  the  switch.  Too  long  a  delay  will  defeat  the  time  saving  purpose  of 
eye  controlled  switching  and  will  be  frustrating  to  the  pilot. 

Assuming  that  the  target  envelope  will  not  subtend  more  than  about  2 
degrees,  a  requirement  of  20  msec  within  the  target  envelope  would  be 
sufficient  to  avoid  confusion  with  saccades.  The  period  must,  however, 
Include  at  least  two  or  three  data  samples  to  make  sense  because  a 
single  sample  contains  no  rate  information.  Depending  on  the  eye 
tracker  update  rate,  this  may  require  more  than  20  msec.  Calhoun  et  al . 
(ref.  25)  required  that  2  out  of  3  successive  samples  from  a  60  Hz  eye 
tracker  be  within  about  2  degrees  visual  angle  from  the  center  of  the 
swi tch. 


If  samples  are  independent  and  normally  distributed  with  standard 
deviation  o  ,  then  the  average  of  n  samples  is  also  a  normal 
distribution  with  standard  deviation  o /{rT. 
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To  be  completely  imperceptible,  the  delay  between  fixation  and 
feedback  would  have  to  be  less  than  50  msec,  but  any  delay  of  120  msec 
or  less  should  be  minimally  annoying.  The  total  delay  between  fixation 
of  a  switch  and  feedback  from  the  display  is  the  sum  of  several 
components;  the  minimum  time  eye  position  is  required  to  be  within  the 
switch  envelope  before  switch  activation,  the  eye  tracker  transport 
delay,  any  lag  caused  by  time  averaging,  time  required  for  the  processor 
to  compare  the  eye  position  data  with  switch  positions,  and  time 
required  for  the  display  system  to  generate  the  appropriate  highlight  or 
cursor.  The  last  two  items  will  probably  require  no  more  than  2  display 
update  intervals.  Since  the  display  update  requirement  is  60  Hz  (ref. 

24) ,  this  delay  translates  to  about  33  msec. 

An  eye  tracker  update  rate  of  120  Hz  with  a  25  msec  transport  delay, 

a  requirement  of  3  consecutive  fields  within  the  target  envelope,  and  a 

3  field  averaging  window  would  add  75  msec  of  delay  and  produce  a  total 

feedback  delay  of  just  over  100  msec.  A  60  Hz  (50  msec  transport  delay) 

eye  tracker  with  the  same  3  field  "on  target"  requirement  and  averaging 
window  would  result  in  a  feedback  delay  of  about  180  msec. 

Once  a  switch  is  "activated"  and  feedback  is  received  by  the  pilot, 
another  300  -  500  msec  will  probably  be  required  for  confirmation.  Data 
reported  by  Calhoun  et  al .  (ref.  26)  indicate  that  on  the  average,  about 
400  msec  was  required  for  use  of  a  joystick  mounted  consent  switch. 

Total  time  from  fixation  of  the  switch  by  the  pilot  to  pilot 
confirmation  of  the  selection  should  range  from  400  -  700  msec  for  a  120 
Hz  eye  tracker.  Dropping  the  update  to  60  Hz  would  add  only  about  80 
msec  to  the  process.  Increasing  the  update  rate  beyond  120  Hz  would  not 
Improve  response  time  by  more  than  about  50  msec.  Considering  these 
factors,  the  eye  controlled  switch  task  does  not  require  higher  than  120 
Hz  eye  tracker  update,  and  would  probably  be  sufficiently  effective  with 
a  60  Hz  system. 

Time  required  to  look  to  the  switch  from  some  other  visual  task 
depends  on  task  loading  and  geometrical  considerations  that  cannot 
easily  be  estimated.  The  same  is  true  of  time  required  to  reach  for  a 
manual  switch.  For  comparisons  of  different  switching  techniques  it  is, 
of  course,  the  total  switching  time,  as  measured  by  Calhoun  et  al .  (ref. 

25) ,  that  is  Important.  In  terms  of  eye  tracker  specifications,  the 
important  point  is  that  eye-controlled  switching  times  cannot  be 
reduced  significantly  by  using  extremely  high  update  rate  eye  trackers. 

The  placement  of  virtual  image  switches  within  the  display  will  be 
an  important  parameter  for  eye-controlled  switching  performance.  Eye 
tracker  range  will  probably  be  slightly  less  than  the  size  of  the 
display.  Maximum  eye  tracker  accuracy  will  probably  be  within  about  20 
degrees  of  visual  iield  center  and,  therefore,  best  eye  controlled 
switching  pe'-Tormanrs  will  be  obtained  if  switches  are  in  this  region. 

Assuming  that  switch  images  will  be  stabilized  with  respect  to 
cockpit  coordinates,  pilots  will  have  to  move  their  heads  enough  to 
place  the  switch  images  in  the  central  region  of  the  display  for  best 
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performance.  Alternately,  switch  separation  and  the  corresponding 
target  envelope  can  be  Increased  or  head  position  fine  tuning  (see 
section  4.3.2)  can  be  used  when  switches  are  in  eccentric  positions  with 
respect  to  the  head. 

The  closer  the  switches  are  to  the  display  center,  the  shorter  the 
distance,  on  average,  that  the  eye  will  have  to  travel  to  fixate  a  given 
switch.  Of  course,  it  will  also  be  important  for  other  display  features 
to  be  near  the  display  center,  and  these  considerations  will  have  to  be 
balanced  when  designing  the  layout. 

5.4  Cueing 

Eye  line-of-gaze  measurements  can  be  used  to  facilitate  conmunication 
of  target  positions  or  other  spatial  position  information  between  pilot 
and  copilot  (backseater) ,  two  pilots,  or  a  pilot  and  the  ground.  For 
example,  assuming  that  both  the  pilot  and  copilot  are  equipped  with  a 
helmet  mounted  virtual  panoramic  display  and  eye  tracker,  the  display 
system  could,  when  appropriate,  display  the  copilot's  poi nt-of-gaze  to 
the  pilot  and  vice  versa.  Such  a  display  should  probably  be  in  the  form 
of  a  highlighted  area  or  outline  subtending  at  least  3  or  4  degrees 
visual  angle  so  that  it  will  be  easily  noticeable  with  peripheral  vision. 

The  system  could  also  display  to  the  pilot  an  arrow  pointing  the 
direction  and  providing  a  distance  cue  to  the  copilot's  po1nt-of-regard 
(or  vice  versa).  This  type  of  communication  may  prove  to  be  far  more 
efficient  than  verbal  descriptions.  Verbal  conmunlcation  would  still 
presumably  be  required  for  one  person  to  indicate  to  the  other  that 
there  is  something  demanding  visual  attention. 

It  should  be  necessary  to  indicate  target  position  only  to  within  a 
parafoveal  distance  of  2  or  3  degrees.  For  a  target  that  is  stationary 
with  respect  to  the  visual  field,  this  implies  a  very  mild  combined 
eye/head  tracker  confidence  interval. 

3  deg  >  JaCC^  +  PREC^  +  (7) 

where  is  a  95%  confidence  interval  for  the  head  tracker,  as 
discusseo  in  section  4.2. 

If  a  moving  target  is  being  tracked,  there  will  be  additional 
uncertainty  due  to  the  characteristics  of  visual  smooth  pursuit  (see 
section  3.2),  but  we  cannot  specify  smooth  pursuit  error  dependably 
enough  to  Include  this  In  the  eye  tracker  specifications.  We  must 
simply  realize  that,  for  physiological  reasons,  point-of-regard  cueing 
may  be  somewhat  less  dependable  when  the  target  is  moving  rapidly 
across  the  visual  field. 

Transport  delay  or  lag  in  the  eye  tracker  and  display  system  will 
cause  position  errors  during  smooth  pursuit  or  compensatory  eye  motions. 
Eye-slaved  cueing  probably  makes  sense  for  target  velocities  up  to  about 
25  deg/sec.  Delay  errors  will,  therefore,  reach  maximum  values  of  25 
deg/sec,  multiplied  by  the  delay.  Assuming  a  maximum  eye  tracker  delay 
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of  three  eye  tracker  update  periods,  60  Hz,  120  Hz,  and  240  Hz  eye 
trackers  would  contribute  maximum  errors  of  1.25  deg,  0.625  deg,  and  0.3 
deg  respectively.  If  we  again  assume  a  33  msec  delay  in  the  display 
system,  then,  for  a  60  Hz  eye  tracker,  we  would  have  to  reduce  the  3 
degree  value  in  equation  (6)  to  about  1.5  degrees.  As  discussed  in 
section  4.3.3,  the  delay  error  can  also  be  reduced  by  appropriate  lead 
compensation. 

5.5  Eye-Controlled  Aiming  and  Target  Designation 

Ballistic  weapons  tend  to  have  a  projectile  spread  of  about  5  feet 
per  thousand,  i.e.,  a  little  over  0.25  degree.  Most  effective  use, 
therefore,  requires  aiming  accuracy  of  about  0.125  degree.  Super 
cockpit  requirements  listed  in  ref.  1  include  eye  aiming  accuracy  of  2 
milliradians  (0.1  degree),  which  is  consistent  with  ballistic  aiming 
requirements.  This  accuracy  is  well  within  the  capability  of  people  to 
align  two  or  more  visual  images,  as  in  traditional  aiming  tasks. 
Unfortunately,  as  discussed  in  section  3,  people  do  not  reliably 
position  their  eye  visual  axis  with  the  same  accuracy.  Even  if  an 
eye/head  tracking  system  could  determine  the  eye  visual  axis  direction 
without  error,  in  the  case  of  a  stationary  target  it  would  not  be 
reasonable  to  expect  a  95%  confidence  interval  for  the  eye  visual  axis 
to  be  smaller  than  0.3  degree  visual  angle  from  the  target.  This  is 
especially  true  if  we  are  considering  a  wide  range  of  subjects  in  a 
combat  or  simulated  combat  environment.  For  a  target  that  is  moving  at 
more  than  10  deg/sec  with  respect  to  the  visual  field,  the  confidence 
interval  is  even  larger. 

There  are  at  least  two  ways  in  which  this  type  of  eye  aiming  task 
could  be  accomplished.  The  first  is  magnification  of  the  target  and 
surrounding  area  to  reduce  the  eye  pointing  accuracy  requirement. 
Magnification  is  possible  if  the  pilot  is  looking  at  a  virtual  image 
created  with  information  from  some  other  sensor,  but  not  if  the  target 
is  being  viewed  directly.  The  second  method  is  to  use  head  movement  for 
fine  adjustment,  as  described  in  section  4.3.2. 

Target  designation,  on  the  other  hand.  Implies  directing  other 
weapon  sensor  systems  (e.g.,  radar,  heat  seekers,  etc.)  to  within  the 
weapon  launch  envelopes.  A  95%  confidence  interval  of  1  degree  is 
probably  sufficient  for  this  type  of  target  designation,  and  eye- 
controlled  target  designation  is  a  feasible  task.  It  offers  the 
significant  advantages  of  potentially  faster  target  acquisition  and  the 
capability  to  function  in  high  g  environments,  where  even  head  motion 
can  be  difficult  and  Inaccurate. 

The  exact  protocol  for  eye-controlled  target  designation  should  be 
the  subject  of  significant  simulator  research,  but  will  probably  be 
similar  in  some  respects  to  that  for  switch  selection.  The  pilot  will 
attempt  to  fixate  the  target.  Feedback  on  the  helmet  mounted  display, 
in  t<ie  form  of  a  highlighted  circle,  cross  hairs,  etc.,  will  indicate 
the  target  locked  onto  by  the  weapons  system,  and  the  pilot  will  confirm 
and  actually  launch  the  weapon  with  a  consent  switch  or  voice  command. 
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For  a  stationary  target  (with  respect  to  the  visual  field),  based  on  the 
assumptions  discussed  above,  the  accuracy  and  precision  requirement  Is 


1  deg 


ACC^  +  PREC^  +  Cl 


hd 


(8) 


A  threshold  filter  can  be  used  to  avoid  introducing  excessive  delay 
and  still  prevent  precision  errors  (jitter)  from  being  transmitted  to 
the  weapon  sensor  servo.  The  threshold  value  should  be  about  the  same 
as  the  precision.  This  will  effectively  make  system  resolution  equal 
precision. 

Delay  In  target  acquisition  feedback  to  the  pilot  will  depend  on  the 
weapon  sensor  system  dynamics,  the  delay  In  display  generation,  the  eye 
tracker  delay,  and  any  minimum  fixation  period  required  to  lock  on  the 
target  sensing  system.  Once  the  appropriate  feedback  Is  received,  there 
will  probably  be  a  200  to  600  msec  delay  for  pilot  confirmation.  Even 
If  the  eye  tracker  delay  Is  50  msec,  the  maximum  transport  delay  that 
would  probably  be  associated  with  a  60  Hz  update  rate  eye  tracker,  this 
will  not  be  a  very  significant  component  of  the  total  response  time. 

A  weapon  could  potentially  be  fired  at  a  target  moving  up  to  25 
deg/sec  across  the  visual  field.  In  the  case  of  a  target  moving  at  this 
speed,  delay  or  lag  In  the  eye  tracker  and  other  parts  of  the  weapon 
aiming  system  would  produce  a  maximum  aiming  err'r  equal  to  25  deg/sec 
times  the  total  delay  in  seconds. 

Eye  trackers  with  3  update  interval  delays  and  update  rates  of  60 
Hz,  120  Hz,  and  240  Hz,  would  contribute  maximum  errors  of  1.25  degrees, 
0.625  degrees,  and  0.3  degrees  respectively.  The  other  processing 
delays  In  the  system  will  probably  contribute  at  least  as  much  error  as 
the  60  Hz  (50  msec  delay)  eye  tracker. 

It  will  certainly  be  advantageous  to  minimize  the  eye  tracker  delay 
component,  but  lead  compensation  will  be  necessary  anyway  If  the  system 
Is  to  be  successful  against  targets  moving  up  to  25  deg/sec,  and  a  60  Hz 
eye  tracker  would  probably  be  sufficient. 

5.6  Pilot  State  Monitoring 

Eye  position  Information  can  potentially  aid  In  determination  of 
pilot  Intent,  appropriateness  of  pilot  actions,  and  the  pilot's 
physiological  state.  Eye  tracking  measurements  can  aid  In  the  first  two 
categories  by  telling  the  pilot  state  monitoring  system  where  the  pilot 
has  been  looking.  Visual  behavior  can  then  be  evaluated  by  the  system 
In  terms  of  the  current  situation. 

This  type  of  evaluation  can  potentially  be  performed  in  near  real 
time,  but  only  with  a  time  constant  that  Is  long  compared  to  average 
fixation  durations,  since  sensible  evaluation  of  pilot  visual  scanning 
behavior  can  be  made  only  by  evaluating  a  series  of  fixations.  For 
determining  pilot  Intent  or  appropriateness  of  pilot  behavior,  eye 
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tracker  performance  requirements  are  essentially  the  same  as  those  for 
display  evaluation  (section  5.2). 

Eye  movements  provide  several  potential  clues  to  physical  and  mental 
state  although,  in  many  cases,  the  relationships  are  not  yet  well 
understood . 

The  most  obvious  physiological  clue  involves  not  eye  movement,  but 
rather  eye  recognition  by  the  eye  tracker.  If  the  eye  tracker  does  not 
detect  the  eye  for  a  period  significantly  longer  than  the  time  required 
for  a  blink  (typically  200  msec),  then  there  is  either  an  eye  tracker 
mtlfuncticn  or  the  pilot's  eyes  are  closed,  perhaps  due  to 
unconsciousness  or  sleep. 

?upil  diameter,  usually  computed  by  an  eye  tracker  as  a  by-product 
of  eye  position  determination,  is  clearly  affected  by  alertness, 
fatigue,  stress,  and  work  load,  among  other  things.  These  effects  can 
best  be  demonstrated  using  very  careful  laboratory  controls  for  target 
luminance  and  other  variables  that  affect  pupil  diameter.  This  often 
involves  averaging  over  many  trials  and  normalizing  with  control  trials. 
For  these  reasons,  it  is  not  yet  clear  whether  pupil  diameter  will  be 
reliable  enough  to  aid  in  pilot  state  monitoring.  At  the  very  least, 
however,  pupil  diameter  variations  should  be  the  subject  of  further 
research  in  the  virtual  cockpit,  and  pupil  diameter  data  should  be 
available  from  the  eye  tracker  subsystem. 

Tole  et  al .  (ref.  27)  and  Ephrath  et  al .  (ref.  28)  have  shown  that 
the  entropy  or  disorderl iness  of  pilot  instrument  scanning  behavior,  as 
measured  by  Markov  chain  analysis,  increases  dramatically  after  a 
certain  work  load  or  stress  threshold  is  reached.  The  threshold  value 
seems  to  increase  as  a  function  of  pilot  skill.  This  phenomenon  would 
also  require  further  research  for  meaningful  use  by  the  super  cockpit 
system.  Eye  tracker  performance  requirements  would,  again,  be  similar 
to  those  for  display  evaluation. 

There  is  some  evidence  that  peak  saccadic  velocities  and  velocity 
profiles  change  when  subjects  become  psychologically  or  physically 
fatigued.  Velocities  as  a  function  of  saccade  amplitude  may  tend  to 
decrease  and  the  velocity  profiles  may  become  more  skewed  (ref.  29  & 

30).  Measurement  of  this  phenomenon  would  have  a  significant  effect  on 
eye  tracker  specifications,  since  a  240  Hz  update  rate  would  probably  be 
required  to  measure  saccadic  velocity.  This  is  the  only  potential 
application  so  far  discusseci  that  very  clearly  requires  an  update  rate 
significantly  faster  than  60  Hz. 

5.7  Consideration  of  Binocular  Eye  Tracking 

Eye  tracking,  in  theory,  can  be  performed  on  either  one  eye 
(monocular  eye  tracking)  or  on  both  eyes  (binocular  eye  tracking). 
Binocular  eye  tracking  will  require  more  helmet  mounted  optical 
components  and  more  processing  capacity  than  will  a  monocular  system 
and,  therefore,  should  be  used  only  if  significant  benefits  result. 
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The  VPD  display  will  be  binocular  and  will  probably  include  partial 
overlap  of  the  monocular  fields.  The  candidate  display  systems 
considered  by  Kocian  in  reference  24  all  have  overlap  sections  of  30  or 
40  degrees. 

When  a  pilot  fixates  a  display  element  outside  of  the  overlap 
section  of  the  display,  that  element  will  be  seen  by  only  one  eye.  The 
appropriate  eye  accommodation  will  occur  to  focus  the  element  being 
fixated  and  the  accommodative  convergence  reflex  (ref.  31)  will  result 
in  the  amount  of  convergence  normally  associated  with  that  amount  of 
accommodation.  In  other  words,  the  eye  not  receiving  the  image  will, 
nonetheless,  point  in  the  appropriate  direction.  Monocular  eye  tracking 
should,  therefore,  permit  accurate  poi nt-of-gaze  measurements,  even  on 
nonoverlapped  sections  of  the  display  visible  only  to  the  nontracked 
eye. 


Binocular  tracking  can,  however,  be  used  to  extend  the  total 
measurable  field  by  biasing  the  measurable  field  in  different  directions 
for  each  eye.  This  possibility  is  explained  more  thoroughly  in  section 
6.2  (optical  path  design).  The  two  monocular  measurement  fields  will 
overlap,  even  if  differentially  biased,  and,  in  these  areas,  binocular 
measurements  could  be  used  to  increase  dependability  by  providing  some 
redundancy. 

Binocular  eye  tracking  would  permit  a  measurement  of  vergence  and 
thereby  an  indication  of  apparent  range  (distance  from  the  observer)  of 
the  fixation  point.  The  effective  resolution  of  the  range  measurement 
will  depend  on  the  uncertainty  in  the  eye  tracker  measurement  as  well  as 
the  range  value.  Figure  5.2  shows  the  convergence  angles  (difference  in 
right  eye  and  left  eye  azimuth  angles)  that  would  be  required  to  exactly 
fixate  a  target  at  different  distances  directly  in  front  of  a  subject. 
Note  that  as  the  target  distance  gets  larger,  the  corresponding 
convergence  angle  rapidly  becomes  very  small.  A  very  small  error  in 
vergence  angle  can,  therefore,  represent  a  very  large  error  in  target 
distance . 

The  candidate  displays  considered  by  Kocian  (ref.  24)  all  use  a 
collimated  display,  so  that  displayed  imagery  appears  at  optical 
infinity.  A  collimated  display  will  allow  a  pilot  to  switch  attention 
between  the  display  imagery  and  an  out-of-cockpit  scene  without  eye 
refocusing.  Vergence  information  from  a  binocular  eye  tracker  could 
potentially  be  used  to  distinguish  between  fixation  on  some  close  object 
within  the  cockpit,  and  a  helmet-mounted  display  element  that  appears  at 
the  same  position  In  the  instantaneous  visual  field,  but  which  is 
focused  at  infinity.  This  should  apply  primarily  to  display  symbology 
that  moves  with  the  head.  Imagery  that  is  stabilized  with  respect  to 
the  cockpit  will  probably  be  positioned  so  as  not  to  appear  superimposed 
on  important  displays  within  the  cockpit.  In  the  super  cockpit,  as 
envisioned  by  Furness  (ref.  32),  all  displays  will  be  virtual  images  and 
the  pilot  may  actually  have  little  reason  to  look  at  physical  objects 
within  the  cockpit. 
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Stereopsis  effects  (perceptions  of  depth)  can  be  produced  in  the 
display  overlap  sections  by  displaying  the  monocular  Images  from 
different  angles,  in  other  words,  by  simulating  binocular  disparity.  If 
stereopsis  is  exaggerated,  correspondingly  exaggerated  eye  vergence 
changes  will  result.  There  will  also  be  a  conflict  between  accotimodation 
cues  and  binocular  disparity  cues,  and  it  is  not  yet  known  whether  such 
cue  conflict  will  cause  any  problems. 

If  the  VPO  employs  a  large  stereopsis  effect,  then  vergence 
measurements  from  a  binocular  eye  tracker  could  conceivably  be  used  to 
distinguish  between  fixation  of  scene  elements  that  are  separated 
primarily  by  apparent  depth.  There  currently  is  not,  however,  a  good 
consensus  on  just  how  stereopsis  will  be  used  by  the  VPO,  if  used  at 
al  1 . 


The  potential  advantages  of  binocular  eye  tracking  are  increased  eye 
tracking  range,  some  measurement  redundancy,  and  the  use  of  vergence 
data  to  measure  apparent  fixation  point  distances  from  the  subject  (at 
least  for  close  distances).  The  drawbacks  are  increased  complexity,  an 
increase  in  the  amount  of  helmet-mounted  equipment,  increased  processing 
requirements  and  an  associated  increased  cost.  These  drawbacks  cannot 
be  assessed  quantitatively  until  eye  tracker  optical  and  processor 
designs  for  the  VPO  are  more  complete.  Furthermore,  potential 
advantages  cannot  be  properly  assessed  until  more  is  known  about  how  the 
VPO  wil 1  be  used. 

Binocular  eye  tracking  is  probably  not  essential  for  any  of  the 
applications  discussed  in  sections  5.2  through  5.6  and  it  is,  therefore, 
advised  that  the  initial  emphasis  be  on  development  of  a  suitable 
monocular  system.  The  need  for  binocular  eye  tracking  should  be 
reassessed  when  both  the  VPO  and  monocular  eye  tracking  designs  are  more 
complete. 

5.8  Ideal  Performance  Specifications 

The  following  set  of  helmet-mounted  eye  tracker  performance 
specifications  would  be  ideal  to  meet  the  virtual  cockpit  requirements 
discussed  in  sections  5.1  through  5.6.  Accuracy  and  precision  values 
correspond  to  the  definitions  in  section  4.2.  The  resolution  value  was 
simply  chosen  to  be  less  than  precision.  For  the  reasons  discussed  in 
section  5.7,  a  monocular  eye  tracker  is  assumed.  In  evaluating  these 

Range:  +40  degrees  Horizontal 
+30  degrees  Vertical 

Accuracy:  .7  degree 

Precision:  .35  degree 

Resolution:  .25  degree 

Update  Rate:  240  Hz 
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Transport  Delay;  3  update  intervals 

Pupil  diameter  range:  2-8  mm 

Optics  Adjustment;  Requires  1  min.  or  less  after  VPO  helmet  donned 

Calibration  Procedure;  -  Requires  1  min.  or  less 

-  Can  be  self-administered  by  pilot 

-  Quick  offset  adjustment  capability 
available  after  calibration 

Tolerance  for  Helmet  Slippage:  4  mm  at  the  eye 

specifications,  it  must  be  remembered  that  accuracy  and  precision 
correspond  to  relatively  stringent  95%  confidence  intervals  as  opposed 
to  the  much  looser  "expected  values",  standard  deviation  values,  or  rms 
values  often  used. 

It  will  probably  not  be  possible  to  meet  all  of  the  above 
specifications,  and  some  compromises  will  have  to  be  made  in  task 
requirements.  Most  noteably,  the  vertical  range  probably  cannot  be 
extended  all  the  way  to  -30  degrees  for  all  subjects  and  0.7  degree 
accuracy  may  not  be  possible  over  more  than  a  20  degree  radius  field. 
This  will  mean  that,  for  best  performance,  pilots  will  have  to  move 
their  heads  enough  to  bring  visual  targets  to  the  central  20  degree 
region  of  the  visual  field.  In  other  cases,  small  compromises  in  task 
requirements  may  signficantly  ease  some  performance  specifications.  For 
example,  if  we  were  to  forego  the  possibility  of  saccadic  velocity 
iiieasurements  for  pilot  state  monitoring,  a  60  Hz  measurement  update  rate 
might  be  sufficient.  The  means  for  best  achieving  desired  performance, 
and  possible  performance  tradeoffs,  are  discussed  in  more  detail  in  the 
following  sections. 


6.0  VPO  EYE  TRACKER  DESIGN  APPROACH 


The  following  sections  outline  a  design  approach  for  a  VPD  eye 
tracker.  The  goal  is  to  come  as  close  as  is  reasonable  to  the  ideal 
specifications  derived  in  section  5. 


The  most  important  requirement  for  the  virtual  cockpit  application 
is  dependability  and  robustness.  In  a  complex  environment,  like  the  one 
in  question,  the  signal  available  to  the  eye  tracker  will  often  be  less 
than  ideal  no  matter  what  eye  tracking  method  is  used.  It  is  essential 
that  the  system  be  "smart"  enough  to  handle  the  "exception  cases"  that, 
in  our  experience,  can  make  or  break  a  system  of  this  type.  It  is  also 
essential  that  the  system  can  be  modified  to  handle  unexpected  problems 
that  will  inevitably  arise  when  it  is  actually  interfaced  to  the  VPD. 


It  is  proposed  that  the  system  employ  the  pupil -to-corneal  reflex 
(CR)  technique  augmented  with  pupil  tracking  alone  at  eccentric  eye 
positions  beyond  the  CR  detection  range.  Since  the  next  generation  VPD 
design  has  not  yet  been  selected,  eye  tracker  optical  path  design  is 
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somewhat  speculative,  and  is  discussed  with  examples  based  on  the  "off 
aperture"  and  "diial  mirror"  candidate  VPD  designs  described  by  Kocian 
(ref.  24).  Similar  principles  apply  to  designs  for  other  possible  VPD 
systems.  For  the  reasons  explained  in  section  5.7,  it  is  recommended 
that  an  initial  eye  tracker  system  be  monocular,  and  it  is  proposed  that 
a  two-dimensional  solid-state  array  device  be  used  as  the  eye  tracker 
sensor.  Algorithm  development  and  processor  system  design  are  discussed 
with  an  emphasis  on  performance  robustness  and  compatibility  with  the 
virtual  cockpit  processor  architecture. 

As  shown  in  the  general  block  diagram  of  figure  6.1,  the  suggested 
system  includes  helmet-mounted  optics,  detector,  and  illuminator 
components;  off-helmet  camera  electronics  and  processing  system, 
peripherals  for  development  and  trouble  shooting,  and  displays  for 
feedback  to  researchers  and  simulator  operators.  Design  choices  and 
various  subsystems  arc  discussed  in  detail  in  subsequent  sections.  In 
one  area  we  have  proposed  a  staged  development  that  will  satisfy  the 
more  important  performance  requirements  first,  followed  by  less 
essential  features.  Specifically,  we  have  suggested  a  60  Hz  prototype 
designed  to  easily  accommodate  subsequent  enhancement  to  higher  update 
rates. 

6.1  Selection  of  Pupil -to-Corneal  Reflex  Technique 

As  described  In  Volume  I  (Review  of  Current  Eye  Movement  Measurement 
Technology),  current  eye  tracking  techniques  can  be  categorized  as 
el ectro-oculographic ,  contact  lens,  and  optical  techniques.  Optical 
techniques  can,  in  turn,  be  divided  into  methods  that  detect  single 
features  on  the  eye,  methods  that  detect  differential  motion  of  two 
features,  eyelid  tracking,  and  laser  doppler  velor.imetry. 

Potential  techniques  for  integration  with  the  VPD  can  be  very 
quickly  narrowed  to  dual  feature  optical  techniques.  The  scleral  coil 
contact  lens  method  must  be  quickly  eliminated  as  too  invasive,  and 
electro-oculography  must  be  eliminated  because  of  the  associated  dc 
drift  problem  (see  sections  2  and  3  of  volume  I).  Eyelid  tracking  is 
not  nearly  accurate  enough,  and  it  only  works  in  the  vertical  axis. 

Laser  velocimetry  can  be  used  to  determine  position  only  by  integrating 
velocity  and  could,  therefore,  be  used  only  in  conjunction  with  a 
separate  position  measurement  technique  needed  to  correct  integration 
errors  and  reset  position  after  blinks.  Single  feature  optical 
detection  techniques  all  have  relatively  large  errors  when  the  detector 
moves  with  respect  to  the  head.  The  helmet  slip  error  associated  with  a 
limbus  tracking  technique,  for  example,  is  over  4  degrees  for  1  mm  of 
optics  translation  with  respect  to  the  head  (see  section  4.1  of  volume 
1).  All  other  single  feature  techniques  will  have  even  larger  helmet 
slip  errors  and  the  VFD  eye  tracker  must  maintain  better  performance  in 
the  presence  of  up  to  4  mm  translation. 

The  dual  feature  techniques  that  have  been  used  successfully  are  the 
pupil -to-corneal  reflex  technique  and  the  dual  Purkinje  image  technique. 
The  dual  Purkinje  image  method  is  by  far  the  more  precise  of  the  two 
and,  as  implemented  by  SRI  (ref.  33),  can  be  used  to  t.i.iasure  the  miniature 
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Figure  6.1  VPD  eye  tracker  block  diagram. 
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drifts  and  microsaccades  that  occur  within  fixations.  The  precision  and 
accuracy  requirements  listed  in  section  5.7  could  be  more  than  satisfied 
by  a  similar  device.  The  SRI  device,  however,  has  optics  that  are  far 
too  bulky  to  be  helmet  mounted,  has  a  range  of  only  about  +10  degrees 
visual  angle,  and  requires  large  pupil  diameters.  With  currently 
available  technology,  we  cannot  envision  a  successful  dual  Purkinje 
image  system  design  that  would  be  suitable  for  integration  with  the  VPD. 

The  pupil -to-CR  technique  can  be  implemented  with  relatively  small 
optics  and  lends  itself  to  helmet  integration.  Current  systems  do  not 
quite  meet  the  accuracy  and  precision  specifications  listed  in  section 
5.7,  but  come  close  enough  that  these  specifications  should  be 
reachable.  The  range  of  the  pupil-to-CR  technique,  which  is  limited 
primarily  by  the  CP.  measurement,  is  at  least  as  great  as  any  optical 
technique  except  for  pupil  tracking  alone. 

Since  a  pupil  center  measurement  is  a  by-product  of  the  pupil-to-CR 
technique,  the  pupil  alone  can  be  used  to  extend  the  range,  although 
with  less  than  maximum  accuracy.  The  update  rate  used  by  most  current 
pupil-to-CR  eye  trackers  is  60  Hz.  A  240  Hz  update  rate  is  definitely 
possible,  although  it  will  require  some  extra  development  to  create  a 
small  enough  sensor  package  and  it  may  make  some  other  performance 
requirements  more  difficult  to  meet. 

In  short,  the  pupil-to-CR  technique,  possibly  augmented  by  pupil 
only  measurements  at  eccentric  regions  of  the  visual  field,  can 
potentially  come  the  closest  to  the  ideal  requirements  outlined  in 
section  5.  Therefore,  we  propose  development  of  an  eye  tracker  based  on 
the  pupil-to-CR  method  for  integration  with  the  VPD. 

6.2  Optical  Path  Design 

Generic  bright  and  dark  pupil  optical  designs  are  shown  on  figures 
6.2  and  6.3.  Note  that  the  dark  puoil  illumination  beam  passes  through 
one  less  beamsplitter  and,  other  things  being  equal,  would  irradiate  the 
eye  with  about  twice  as  much  light  as  the  bright  pupil  illuminator  beam. 
The  size  of  the  CR  will  be  proportional  to  the  angle,  at  the  eye, 
subtended  by  the  illuminator  diffuser.  The  illuminator  collector  lens, 
field  lens,  and  oistance  between  them  are  determined  by  the  size  of  the 
diffuser  that  is  to  be  "filled"  and  the  diffuser-to-eye  distance. 

At  ASL,  we  have  found  that  this  type  of  illuminator  design  usually 
provides  sufficient  illumination  while  remaining  well  below  eye  safety 
limits,  as  defined  by  Sliney  and  Wolbarsht  {ref.  34).  Depending  on  the 
specific  configuration,  irradiation  at  the  eye  is  usually  between  0,1 
and  0.5  mW/cm^.  If  more  power  is  needed,  because  the  beam  may  have  to 
pass  through  VPD  optical  elements  Oi  oecause  of  long  optical  path 
lengths,  then  the  LED  can  be  replaced  with  a  more  powerful  incandescent 
source  relayed  to  the  helmet  through  a  fiber  optics  cable.  Alternately, 
the  diffuser  can  be  removed  and  the  source  optically  magnified  to 
subtend  the  necessary  visual  angle. 
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Figure  6.2  Basic  bright  pupil  eye  tracker  optics. 

The  eye  camera  field-of-view  should  include  the  entire  pupil  during 
the  worst  case  situation  of  maximum  pupil  diameter  (8  mm),  maximum  eye 
rotation  of  interest  (40  degrees  horizontal),  and  maximum  helmet  slip  (4 
trni).  These  values  require  a  horizontal  camera  field  of  about  27  mm  at 
the  eye. 

A  bright  pupil  optics  system  has  a  single  central  axis  for  both  the 
detector  camera  and  illumination  optical  paths.  A  dark  pupil  imaging 
system,  on  the  other  hand,  must  have  separate  illumination  and  detector 
paths.  Even  in  the  case  of  a  dark  pupil  system,  range  will  be  maximized 
by  keeping  the  two  optical  paths  within  several  degrees  of  each  other. 

We  will  define  the  central  axis  as  the  eye  visual  axis  when  the 
pilot  looks  at  the  center  of  the  VPD  field.  In  the  vertical  plane,  the 
illuminator  and  the  detector  optical  axes  should  intersect  the  eye  from 
below  the  central  axis  in  order  to  minimize  occlusion  of  the  pupil  by 
the  upper  eyelid.  Experience  at  ASL  indicates  that  vertical  range  with 
respect  to  the  central  axis  will  be  maximized  If  the  eye  tracker  optical 
axis  is  10  to  15  degrees  below  the  central  axis.  The  resulting  range  is 
usually  about  +15  to  +20  degrees  with  respect  to  the  central  axis  for 
unoccluded  CR  recognition  and  about  +30  degrees  to  -20  degrees  for  a 
reasonably  unoccluded  pupil. 
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Figure  6.3  Basic  dark  pupil  eye  tracker  optics. 

In  the  horizontal  plane,  range  about  the  central  axis  will  be  most 
symmetrical  when  the  Illumination  and  detector  axes  are  biased  to  the 
temporal  side  of  the  central  axis  by  about  6  degrees  (the  angle  between 
the  eye  optical  and  visual  axes).  The  resulting  horizontal  range  for  a 
measurable  CR  will  usually  be  about  ^ZO  to  +30  degrees. 

If  binocular  eye  tracking  Is  used  as  discussed  In  section  5.7,  It  Is 
possible  to  extend  the  total  range  by  differentially  biasing  the  two 
monocular  fields.  For  example,  assume  that  the  detector  axes  on  a 
binocular  eye  tracker  were  biased  11  degrees  to  the  temporal  side  of  the 
central  llne-of-gaze  for  each  eye  (Instead  of  only  6  c agrees  as 
suggested  above).  The  range  for  a  measurable  pupil  ani  CR  would  be 
shifted  5  degrees  to  the  left  for  left  eye  measurements,  and  5  degrees 
to  the  right  for  right  eye  measurements.  The  total  measurable  field 
would  be  extended  by  5  degrees  on  each  side  (to  about  +25  to  +35 
degrees),  but  a  10  degree  band  at  each  periphery  of  thTs  extended  field 
would  be  measurable  with  only  one  eye. 
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Certain  optical  path  choices  may  require  that  the  eye  tracker  look 
through  one  or  more  display  components.  Attenuation  by  such  components 
must  be  considered.  Furthermore,  If  both  the  Illumination  beam  and 
detector  path  pass  through  the  same  optical  surface,  the  specular 
reflection  from  that  surface  may  form  an  image  In  the  detector  fleld-of- 
view,  and  may  interfere  with  detection  of  the  pupil  and  corneal  reflex. 
Anti-reflective  coatings,  even  multicoatings,  usually  do  not  completely 
eliminate  such  reflections.  The  location  of  the  specular  reflection 
image  (a  virtual  image  of  the  illuminator)  Is  determined  by  the  shape  of 
the  reflecting  surface  and  the  Illuminator  beam  angle  of  Incidence. 

This  relation  Is  Illustrated  In  figure  6.4  for  a  spherical  combiner 
surface. 

The  remainder  of  the  optical  path  design  is  dependent  on  the  VPD 
configuration.  In  the  following  subsection,  a  brief  experiment  Is 
described  using  the  off-aperture,  dual  mirror,  and  pancake  window 
breadboard  helmets.  In  the  subsequent  two  subsections,  optical  path 
possibllties  are  presented,  as  examples,  for  the  off  aperture  and  dual 
mirror  systems.  The  optical  path  discussions  apply  equally  to  bright  or 
dark  pupil  eye  tracker  optics. 

Similar  analyses  are  possible  for  other  display  designs,  including 
the  pancake  window  system,  a  holographic  system  made  by  Hughes  (ref. 

24),  a  helmet-mounted  display  made  by  6EC  incorporating  night  vision 
sensors  (ref.  35),  anc  others.  A  rigorous  optical  path  design  and 
prototype  construction  for  a  particular  display  will  require  detailed 
drawings  of  the  display  optics  and/or  a  sample  system  that  can  be 
dismantled  for  experimentation. 

6.2.1  Experiment  with  Breadboard  VPO  Designs 

A  very  brief  experiment  was  performed  with  off  aperture,  dual 
mirror,  and  pancake  window  breadboard  systems  at  the  AAMRL  facility. 
Small  dark  and  bright  pupil  optics  modules  were  hand-held  at  various 
positions  to  obtain  eye  Images  while  a  subject  wore  one  of  the  VPD 
breadboard  helmets.  The  dark  and  bright  pupil  optics  modules  were 
similar  to  those  shown  in  figures  6.2  and  6.3,  except  that  the  hot 
mirror  shown  in  tho^e  figures  was  not  used.  The  detector  was  a 
miniature  video  camera  and  the  field-of-view  at  the  eye  was  about  25  mm 
wide  by  20  mm  high.  The  resulting  Images  were  displayed  on  a  monitor 
and  were  also  recorded  on  a  VHS  video  cassette  recorder. 

With  each  VPD  optics  systems,  stroke,  typical  raster  scene,  and  flat 
field  (uniform  brightness)  raster  displays  were  used  to  provide  data 
under  different  scene  luminance  conditions,  although  luminosity  was  not 
actually  measured.  The  flat  field  raster  displays  represent  an  upper 
bound  to  scene  luminance  for  each  display  with  the  raster  scene 
representing  a  more  typical  case  and  the  stroke  display  representing  a 
minimum  brightness  condition.  All  tests  were  performed  indoors,  mostly 
with  room  lights  off,  and  the  display,  therefore,  provided  virtually  all 
of  the  luminance  In  the  subject's  visual  field.  Only  one  subject  was 
tested  with  each  display.  Whenever  one  of  the  eye  imaging  optics 
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Figure  6.4  Location  of  specular  reflection  image  from  a  spherical 


modules  was  focused  at  a  new  distance,  a  ruler  image  was  later  recorded 
at  the  same  distance  to  provide  a  reference  for  determination  of  pupil 
diameters . 

Figure  6.5  is  a  sketch  showing  the  off  aperture  design  concept. 
Helmet  mounted  projectors  direct  an  image  to  curved  (mildly  aspheric] 
combiners  in  front  of  each  eye  which  act  as  both  mirror  and  lens  for  the 
projected  images.  The  pilot  will  see  both  the  projected  image  and  the 
real  scene  through  the  beam  splitters.  For  the  current  breadboard 
design,  the  display  f1e1d-of*v1ew  is  50  degrees  horizontal  by  37.5 
degrees  vertical  (ref.  24).  The  eye  relief  (minimum  distance  from  the 
eye  to  combiner)  is  about  56  mm. 


Figure  6.5  Off-aperture  VPD  helmet  concept,  from  Kocian  (ref.  24). 

The  dual  mirror  design  is  shown  in  figure  6.6.  The  display  is 
projected  from  below  to  a  combiner/beam  splitter  assetnbly.  The  display 
f1eld-of-v1ew  is  50  degrees  horizontal  by  45  degrees  vertical  with  an 
eye  relief  of  about  it  mm  (ret.  24). 

The  pancake  window  system  (figure  6.7)  reflects  a  projected 
polarized  display  image  from  a  flat  combiner  through  the  pancake  window. 
The  pancake  window,  used  to  create  a  collimated  virtual  image  of  the 
display,  consists  of  a  curved  combiner  plus  a  series  of  flat  combiners. 
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Figure  6.6  Dual  mirror  VPO  design  concept,  from  Koclan  (ref.  24). 

polarizers,  and  quarter  wave  plates.  A  detailed  explanation  of  the 
technique  can  be  found  In  reference  36. 

Pupil  and  corneal  reflection  Images  easily  detectable  to  an  observer 
were  obtained,  with  both  bright  and  dark  pupil  optics,  through  the  off 
aperture  system  curved  combiner,  and  through  the  entire  beam  splitter/ 
combiner  assembly  on  the  dual  mirror  system.  In  the  case  of  the  pancake 
window  system,  a  detectable  Image  could  be  obtained  through  the  pancake 
window  from  the  nasal  side  of  the  flat  combiner;  but  not  through  both 
the  flat  combiner  and  the  pancake  window,  primarily  because  of  multiple 
specular  reflections  In  the  Image. 

In  the  case  of  the  off  aperture  system,  a  suitable  Image  was  also 
obtained  looking  from  Just  below,  rather  than  through,  the  curved 
combiners.  The  geometry  of  the  other  two  systems  (dual  mirror  and 
pancake  window)  did  not  permit  a  clear  view  of  the  eye  without  looking 
through  some  of  the  display  optics  components,  as  previously  described. 

Specular  reflections  from  display  components  created  some 
difficulties  with  both  the  dual  mirror  and  pancake  window  systems. 
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Figure  6.7  Pancake  window  design  concept. 

especially  when  using  the  bright  pupil  optics  module.  !n  the  case  of 
the  dual  mirror  system,  the  specular  reflection  from  the  curved  combiner 
tended  to  cover  the  pupil  unless  the  optical  path  was  well  below  the 
central  1 ine-of-gaze.  Mechanical  obstruction  by  the  lower  helmet 
cowling  prevented  the  hand-held  bright  pupil  optics  module  from  being 
positioned  as  far  below  the  central  llne-of-gaze  as  would  probably  be 
Ideal;  but  when  held  so  that  its  smallest  dimension  was  oriented 
vertically.  It  was  possible  to  position  the  optics  module  so  that  the 
specular  reflection  was  just  out  of  the  camera  field.  With  the  pancake 
window  system,  the  oxygen  mask  assembly  caused  a  similar  mechanical 
restriction.  The  best  bright  pupil  Image  achieved  had  a  large  specular 
reflection  (from  the  pancake  window)  several  millimeters  below  the 
pupil,  but  still  well  within  the  video  Image  field  when  the  pupil  was 
centered.  Because  the  optics  were  hand-held  and  the  helmet  was  not 
stabilized,  actual  optical  path  angles  could  not  be  determined. 

The  dark  pupil  images  were  less  of  a  problem  because  the  noncoaxial 
Illuminator  beam  (see  figure  6.3),  which  is  displaced  by  about  5  degrees 
from  the  detector  path,  could  be  placed  at  larger  angles  with  respect  to 
central  llne-of-gaze  more  easily.  For  example,  when  the  dark  pupil 
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system  camera  had  approximately  the  same  optical  path  with  respect  to 
the  pancake  window  helmet  as  the  bright  pupil  camera,  the  dark  pupil 
illuminator  beam  was  5  degrees  lower  than  the  bright  pupil  beam.  The 
specular  reflection  from  the  pancake  window  was  far  enough  below  the 
dark  pupil  to  remain  out  of  the  camera  field-of-view. 

Thorough  assessment  of  potential  specular  reflection  problems  will 
require  accurate  drawings  of  the  display  components  involved  and  their 
position  relative  to  the  eye.  Alternately,  the  breadboard  display 
optics  must  be  partially  disassembled  so  that  an  optical  bench  can  be 
used  for  precise  empirical  experiments. 

Using  the  off  aperture  display  optics,  pupil  diameters  varied  from 
5.6  to  6.0  mm  with  the  stroke  display,  4.8  to  6.0  mm  with  the  raster 
scene  display,  and  2.6  to  3.0  mm  with  the  flat  field  raster  display. 
Using  the  dual  mirror  optics,  pupil  diameters  were  in  the  range  of  4.5 
to  5.5  mm  with  all  displays  (stroke,  raster  scene,  and  flat  field 
raster).  Using  the  pancake  window  optics,  pupil  diameter  values  were 
5.0  to  5.5  mm  with  the  raster  scene  display  and  4.0  to  5.0  mm  with  the 
flat  field  raster  display.  The  stroke  display  was  not  tested  with  the 
pancake  window  system.  In  all  of  the  above  cases,  both  dark  and  bright 
pupil  images  had  an  easily-discriminable  pupil  and  corneal  reflex. 

Aside  from  potential  specular  reflection  problems,  all  images  were 
probably  suitable  for  automatic  recognition  by  an  eye  tracker. 

Pupil  diameter  can  vary  considerably  between  subjects  and  for  the 
same  subject  at  different  times.  Only  one  subject  was  used  for  each 
test  and  the  test  conditions  were  less  than  rigorous.  Results  from  this 
brief  experiment  must,  therefore,  be  regarded  as  only  a  preliminary 
indication  of  conditions  that  will  be  encountered. 

We  can  conclude  that,  if  necessary,  infrared  light  transmission  is 
probably  sufficient  for  an  eye  tracker  through  the  off  aperture  curved 
combiner,  the  dual  mirror  combiner  assembly,  and  the  pancake  window.  If 
an  eye  tracker  optical  path  must  pass  through  the  dual  mirror  combiner 
assembly  or  through  the  pancake  window,  specular  reflections  may  or  may 
not  present  a  problem.  This  potential  problem  can  be  further 
investigated,  as  previously  described.  The  helmet-mounted  displays 
alone  do  not  appear  to  be  bright  enough  to  reduce  pupil  diameters 
significantly  below  3  mm.  The  effect  of  a  real  daylight  scene  under 
actual  flight  conditions  was  not  considered.  It  Is  not  yet  clear 
whether  dark  or  bright  pupil  optics  will  produce  the  best  image  under 
the  full  range  of  expected  conditions. 

6.2.2  Possible  Eye  Tracker  Optical  Paths  for  Off  Aperture  VPO  Design 

The  off  aperture  design  was  described  In  the  previous  section  and  is 
shown  in  figure  6.5.  The  basic  potential  paths  for  eye  tracker  optics 
are  shown  in  figures  6.8  and  6.9.  Paths  1  and  2  do  not  require  the 
addition  of  optical  surfaces  within  or  near  the  pilot's  fielu--or-view. 
These  paths  are  the  preferred  choices  for  minimum  obtrusiveness  and 
minimum  modification  of  the  current  planeform.  There  are,  however. 
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Figure  6.8  Eye  tracker  optical  path  choices  for  off  aperture  VPD 
helmet. 

a  potential  optical  problems  that  cannot  be  fully  evaluated  without  more 
detailed  knowledge  of  the  display  optics  or  additional  experimentation 
with  a  prototype  device. 

Path  number  1  passes  through  the  display  relay  optics.  The  details 
of  the  relay  optics  are  not  available,  but  since  the  eye  sees  the  CRT 
Image  focused  at  Infinity,  It  follows  that,  with  appropriate  optics,  the 
eye  can  be  Imaged  from  the  position  of  the  CRT.  More  Information  Is 
needed,  however,  to  determine  the  exact  nature  of  such  optics.  The  45 
degree  mirror  In  the  display  path  Just  above  the  CRT  would  have  to  be 
replaced  with  a  "cool  mirror,"  I.e.,  a  mirror  highly  reflective  In  the 
visible  spectrum  for  the  display  and  highly  transmissive  In  the  Infrared 
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Figure  6.9  Additional  eye  tracker  optical  path  choices  for  off  aperture 
VPO  helmet. 

for  the  eye  tracker.  It  Is  probably  possible  to  achieve  a  coating  that 
would  be  adequate  for  both  the  display  and  the  eye  tracker,  but  this 
must  be  experimentally  verified.  An  Infrared  reflective  coating  would 
also  have  to  be  added  to  the  curved  display  combiner.  The  most  serious 
potential  difficulty  Is  that  the  eye  tracker  Illumination  beam  would 
have  to  pass  through  multiple  optical  elements  In  the  relay  assembly  and 
will  form  specular  reflections  on  all  of  them.  At  least  some  of  these 
reflections  would  probably  be  In  a  critical  part  of  the  detector  field 
and,  even  with  antl-reflcctl ve  coatings  on  the  elements,  might  Interfere 
with  eye  tracker  performance. 

The  path  labeled  number  2  would  not  have  the  specular  reflertlon 
problem,  but  some  other  Issues  must  be  considered.  The  curved  combiner 
will  probably  produce  a  magnified  image  of  the  eye  near  the  front  of  the 
relay  optics  assembly.  More  detailed  Information  about  the  combiner  Is 
needed  to  determine  the  exact  properties  of  such  an  Image  and  the  optics 
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necessary  to  obtain  an  unaberrated  eye  Image  at  the  eye  tracker 
detector.  If  detector  optics  need  to  be  large,  this  would  be  contrary 
to  the  goal  of  minimizing  helmet-mounted  equipment.  A  hot  mirror  would 
be  required  at  the  end  of  the  display  relay  optics  to  reflect  the 
infrared  eye  tracker  light  while  passing  the  visible  display  light.  As 
in  the  case  of  path  number  1,  the  curveo  combiner  would  need  an  infrared 
reflective  coating. 

Path  number  3  in  figure  6.9  is  the  closest  to  current  ASL  helmet 
mounted  eye  tracker  optics,  and  Is  the  most  straightforward  in  terms  of 
eye  tracker  performance.  The  eye  tracker  combiner  (hot  mirror)  would, 
however,  be  in  the  path  of  the  display  beam.  Although  it  would  be 
coated  for  maximum  transmittance  and  minimum  reflectance  in  the  visible 
spectrum,  this  beam  splitter  may,  nonetheless,  interfere  with  display 
performance.  The  eye  tracker  beam  splitter  would  also  decrease  eye 
relief  (distance  from  the  eye  to  the  nearest  optical  surface).  By 
increasing  the  optical  path  angle  from  15  to  about  25  degrees  below 
central  line-of-qaze,  the  hot  mirror  could  be  moved  below  the  display 
combiner  and  out  of  the  display  path.  Increasing  this  angle  will 
displace  the  measurable  field-of-view  downward,  and  may  make  the  corneal 
reflex  undetectable  when  the  subject  looks  at  the  top  of  the  display. 

Path  number  4  is  similar  to  3,  but  requires  an  extra  mirror  to  bend 
the  path  around  the  crown  of  the  helmet  so  that  the  camera  and 
illuminator  can  lie  flush  against  the  crown  of  the  helmet.  The  path 
length  is  also  a  bit  longer. 

Path  number  5  would  not  interfere  with  the  display  in  any  way.  The 
primary  disadvantage  to  path  number  5  is  that  it  requires  that  a  hot 
mirror  be  supported  at  a  long  distance  from  the  helmet  center  of  mass. 
This  hot  mirror  would  probably  have  to  be  supported  by  the  lower  helmet 
cowling.  The  path  also  must  pass  through  the  curved  display  combiner 
but,  as  indicated  by  the  test  described  in  section  6.2.1,  this  should 
not  be  a  problem. 

6.2.3  Possible  Eye  Tracker  Optical  Paths  for  Dual  Mirror  VPO  Design 

The  dual  mirror  design  was  described  in  section  6.2.1  and  is  shown 
in  figure  6.6.  Figure  6.10  shows  the  optical  path  that  would  present 
the  fewest  optical  problems  for  implementation  of  an  eye  tracker.  The 
infrared  transmittance  of  the  beam  splitter/combiner  assembly  was  shown 
to  be  sufficient  by  the  test  described  in  6.2.1.  Determination  of  the 
specular  reflection  location  will  require  knowledge  of  the  spherical 
combiner  radius  of  curvature  and  the  precise  location  of  its  center  of 
curvature  with  respect  to  the  eye  (see  figure  6.4). 

A  second  path,  not  shown  in  figure  6.10,  is  also  a  possibility.  The 
flat  element  in  the  beam  splitter/combiner  assembly  could  be  coated  for 
infrared  reflectivity  and  used  to  reflect  an  eye  image  upward.  Because 
of  the  position  and  orientation  of  this  flat  combiner,  the  reflection  of 
a  ray  that  intersects  its  surface  from  below  the  central  line-of-gaze 
would  probably  be  directed  toward  the  pilot's  eyebrows  or  just  obove.  A 
relay  mirror  would,  therefore,  be  required  a  very  short  distance  above 
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Figure  6.10  Possible  eye  tracker  optical  path  for  dual  mirror  VPD 
helmet. 

the  pilot's  eye  or  eyebrow,  followed  by  a  second  mirror,  at  the  top  of 
the  cowling,  to  direct  the  image  back  over  the  forehead. 

6.3  Sensor  Selection 

The  mcst  basic  decision  regarding  the  choice  of  a  detector  is 
whetner  to  use  a  device  that  provides  very  complete  spatial  information, 
but  requires  a  lot  of  processing,  or  a  sensor  that  produces  less 
complete  Information  and  carries  a  smaller  processing  burden.  Different 
types  of  optical  detectors  were  reviewed  in  section  5  of  volume  I 
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(Review  of  Current  Eye  Movement  Measurement  Technology).  We  feel  strongly 

that  the  best  sensor  choice  for  a  VPO  eye  tracker  is  a  full  two-dimensional 

array  capable  of  producing  grey  scale  information.  This  choice  is  driven  j 

by  the  need  to  achieve  a  high  degree  of  dependability  in  an  environment  j 

where  imaging  may  often  be  less  than  ideal.  Continuing  advances  in  high 

speed  digital  processing  are  rapidly  reducing  the  computational  burden 

of  dealing  with  large  numbers  of  image  elements. 

If  thp  i.nage  is  collapsed  onto  linear  arrays  by  cylindrical  lenses 
or  the  equivalent,  high  spatial  frequency  information  is  irretrievably 
lost  and  a  human  readable  image  cannot  easily  be  created.  Other  sensors 
provide  even  less  complete  information. 

Commerci al ly-avai 1 abl e  CCD  cameras  with  miniature  remote  sense  heads 
are  appropriate  for  a  60  Hz  version  of  the  VPD  eye  tracker.  A  typical 
camera  used  in  current  ASL  helmet  mounted  eye  trackers  has  a  510 
Horizontal  x  492  Vertical  pixel  array  (effectvvely  510  H  x  246  V  when 
used  in  a  noninterlaced  mode),  has  a  sense  head  that  measures  32  mm  x  42 
mm  X  25  mm  and  a  small  electronics  unit  (32  mm  x  42  mm  x  118  tnm)  attached 
to  the  sense  head  by  a  flexible  four  foot  cable.  The  remote  sense  head 
is  small  enough  for  helmet  mounting  and,  by  repackaging,  can  be  made 
even  smaller  and  lighter,  if  necessary.  Similar  devices  with  smaller 
sense  heads  will  probably  be  cormercially  available  in  several  months. 

For  a  120  Hz  or  240  Hz  version  of  the  VPD  eye  tracker,  an 
appropriate  detector  cnoice  would  be  the  Reticon  MC9128  camera  with  a 
128  X  128  pixel  array.  The  full  pixel  array  can  be  scanned  at  up  to  300 
Hz.  A  possible  alternate  choice  is  the  CIO  Technologies,  Inc.  (formerly 
GE  Intelligent  Vision  Systems  Operation)  CID2250.  Although  the  CID 
techologies  device  is  a  512  x  512  pixel  array,  at  240  Hz  only  512  x  64 
pixels  can  be  used.  EG&G  Reticon  supplies  a  256  x  256  pixel  device 
(Model  M9256)  but  it  also  can  update  at  over  100  Hz  only  by  scanning  a 
subset  of  the  available  lines.  Since  the  eye  cannot  move  very  far  in 
one  sample  period,  one  of  these  higher  resolution  devices  could 
potentially  be  used  to  advantage  by  always  scanning  a  subset  of  lines 
centered  over  the  last  pupil  position.  If  the  pupil  is  not  found  in 
this  region,  after  a  blink  for  example,  one  longer  sample  period  would 
be  required  to  scan  the  entire  field.  We  estimate  that  128  x  128  pixels 
will  provide  sufficient  resolution  and  we  would  elect  not  to  impose  this 
additional  complexity. 

The  EG&G  Reticon  M9128  device  is  not  sold  with  a  miniature  sense 
head,  and  will  have  to  be  repackaged,  as  would  any  currently  available 
high  speed  array.  Before  solid  state  cameras  were  commonly  available 
with  miniature  sense  heads,  ASL  re-engineered  and  miniaturized 
conventional  60  Hz  solid  state  cameras  for  helmet  mounting,  and  we  are 
confident  that  this  can  be  accomplished.  It  will  also  be  necessary  to 
design  and  build  a  controller  to  provide  the  necessary  timing  and 
control  signals. 

Increased  update  rate  results  in  decreased  camera  integration  time 
and  causes  a  corresponding  decrease  in  sensitivity.  At  ASL,  we  have,  in 
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the  past,  produced  useable  bright  pupil  images  with  a  Reticon  M9128 
operating  at  200  Hz  under  laboratory  conditions.  We  are  confident  that 
enough  sensitivity  can  be  achieved  with  this  device  at  240  Hz,  even 
under  operational  conditions,  for  either  bright  or  dark  pupil  use.  It 
may  be  necessary  to  maximize  sensitivity  by  thermoel ectrical 1y  cooling 
the  sensor  chip  to  decrease  dark  current,  and  by  correcting  for 
individual  pixel  offset. 

We  recommend  that  a  60  Hz  prototype  system  be  developed  first, 
allowing  concentrated  effort  on  the  most  important  problem; 
specifically,  robust  recognition  and  center  computation  in  the  presence 
of  artifacts  and  nonideal  images.  Enhancement  to  240  Hz  will  require 
additional  engineering  as  described  above,  but  will  not  involve 
significant  duplication  of  effort. 

6.4  Algorithm 

Current  ASL  pupil-to-CR  eye  trackers  use  size,  shape,  smoothness, 
and  continuity  criteria  to  recognize  the  pupil  and  corneal  reflection  in 
a  video  image  of  the  eye.  The  recognition  logic  is  capable  of  a  very 
significant  amount  of  artifact  rejection,  and  has  been  used  quite 
successfully  under  a  wide  range  of  conditions.  It  does,  however,  have 
limits  imposed  primarily  by  the  amount  of  available  information,  and  the 
speed  of  the  processor  used.  The  computer  processor  receives  only  edge 
information  determined  by  thresholding  over  horizontal  scan  lines 
through  the  video  image.  Edges  or  portions  of  edges  that  do  not  cross 
the  appropriate  threshold  boundary  in  a  given  video  field  are 
irretrievably  lost. 

The  possibilities  for  recognition  algorithms  are  significantly 
expanded  if  the  entire  grey  scale  video  field  is  made  available  to  the 
processor.  Local  searches  using  different  threshold  values  and  edge 
enhancement  techniques  can  be  used  to  extend  incomplete  information 
generated  by  initial  thresholding.  For  example,  when  a  smooth  edge  is 
prematurely  truncated,  an  edge  enhancement  algorithm  can  be  locally 
employed  to  see  if  the  edge  really  continues.  If  a  limbus  edge  is 
detected  instead  of  the  pupil  edge,  then  an  area  larger  than  a  "legal" 
pupil  will  be  enclosed,  and  the  local  area  within  can  be  searched  with  a 
different  threshold  and  with  edge  enhancement  techniques  to  find  the 
true  pupil  edge.  It  should  be  possible  to  significantly  improve  the 
robustness  of  pupil  and  CR  recognition  over  current  systems. 

Initial  edge  information  will  still  be  generated  with  threshold 
techniques.  Current  systems  require  manual  adjustment  of  thresholds. 

The  VPD  eye  tracker  system  should  choose  threshold  values  automatically 
with  a  histogram  technique. 

Detection  and  recognition  results  in  groups  of  pixels  presumed  to 
belong  to  the  pupil  and  CR.  The  centers  of  the  pupil  and  CR  can  usually 
be  determined  with  a  resolution  and  precision  of  one  pixel  or  better  as 
illustrated  by  the  following  algorithm.  Assumes  an  untruncated  circular 
pupil  image.  To  compute  horizontal  center,  average  the  end  point 
positions  of  horizontal  pixel  lines  included  in  the  pupil.  The  lines 


48 


near  vertical  center  contain  nore  pixels,  and  should  actually  be 
weighted  more  heavily  than  those  near  the  top  or  the  bottom.  If  we 
simply  do  not  Include  the  top  and  bottom  15%  of  the  pupil  lines  in  the 
calculation,  the  remaining  lines  would  have  weights  ranging  only  from 
1.0  to  .707  and  the  weighting  function  can  be  ignored.  Vertical  center 
can  be  computed  by  the  same  procedure  with  respect  to  vertical  lines  of 
pixel s. 

The  resolution,  i.e.,  the  smallest  possible  change  in  the 
measurement,  will  be  1/n  pixels,  where  n  is  the  number  of  end  points 
used.  Since  we  have  one  pixel  quantization,  the  true  position  of  any 
given  line  has  a  uniform  probability  distribution  between  +0.5  pixels, 
and  the  ?ms  error  associated  with  each  end  point  value  wilT  be  on  the 
order  of  0.3  pixels.  If  we  assume  statistical  independence  between  end 
point  errors,  the  averaging  process  will  reduce  thi?  rms  error  by  'TTf 
where  n  is  number  of  end  points  used.  The  algorithm  described  would 
yield  a  pupil  center  computation  with  a  resolution  of  1/n  pixels  and  a 
precision,  defined  by  twice  the  rms  error,  of  Z/VIT.  This  analysis 
considers  only  quantization  errors,  not  video  noise,  physiological 
variability,  and  other  error  sources. 

Eyelid  truncation,  corneal  reflection  occlusion  of  a  pupil  edge,  and 
occlusions  by  other  artifacts  will  have  the  effect  of  reducing  the 
number  of  lire  end  points  available  for  the  computation.  For  example, 
if  the  CR  occludes  one  section  of  pupil  edge  at  the  lower  right  of  the 
pupil,  we  can  exclude  from  the  horizontal  center  computation  the 
horizontal  lines  passing  through  that  section  of  the  pupil  edge.  So 
that  we  do  not  bias  the  result  if  the  pupil  is  actually  a  tilted  elipse, 
we  can  also  exclude  a  similar  set  of  lines  near  the  top  of  the  pupil. 

If  too  few  edge  points  would  be  left,  symmetry  considerations, 
interpolation,  or  a  combination  of  these  techniques  can  be  used  to  fill 
in  the  missing  edge  data. 

The  computation  ii.eusurement  can  be  further  Improved  with  other 
techniques.  Quantization  errors  can  be  further  reduced  by  using  grey 
scale  information  for  interpolation.  If  an  edge  point  along  ore 
horizontal  line  is  at  pixel  n,  meaning  that  pixel  n  is  above  a  threshold 
value  T  and  pixel  r+1  is  below  the  threshold,  then  edge  position  P  can 
be  defined  as 


T-v{n) 

P  =  ri+  -  (9) 

v(n+l)  -  v(n) 


where  v(i)  is  video  level  of  pixel  i. 

The  measurement  can  also  be  improved  by  simple  spatial  filtering  to 
smooth  pupil  edges  and  reduce  both  quantization  errors  and  video  noise 
errors . 

The  eye  line-of-gaze  measurement  will  be  computed  with  the  pupil -to- 
CR  technique  in  approximately  the  central  40-degree  diameter  visual 


49 


field  over  which  the  CR  can  be  detected.  Beyond  the  central  fleld-of- 
view,  where  a  corneal  reflection  cannot  be  detected,  the  measurement 
mode  will  switch  and  eye  position  will  be  determined  by  pupil  center 
only.  Measurement,  with  only  the  pupil,  has  the  disadvantage  of  being 
affected  by  helmet  slippage,  and  this  problem  is  handled  as  follows. 
Whenever  both  pupil  and  corneal  reflection  are  visible,  a  slip  or 
correction  factor  is  computed  to  “reset*'  the  pupil  center  and 
"recalibrate"  the  pupil  map.  Since  helmet  slippage  Is  a  relatively 
infrequent  phenomenon,  this  correction  has  been  found  to  be  satisfactory 
and  is  used  by  current  ASL  helmet-mounted  eye  trackers. 

An  alternate  or  supplementary  technique  Is  the  use  of  pupil 
elllpticity  calculation  at  eccentric  eye  positions.  The  pupil  technique 
is  not  subject  to  helmet  slip  error,  but  requires  substantially  more 
computation  and  will  not  be  used  unless  necessary. 

6.5  Calibration 

Although  we  know  the  underlying  relationship  between  the  pupil -to-CR 
vector  and  eye  llne-of-gaze  (see  section  4.5.1  of  volume  1),  empirical 
data  are  required  to  correct  for  Individual  subject  differences.  The 
required  data  are  gathered  during  a  "calibration"  procedure  which,  to 
ensure  maximum  accuracy,  must  be  performed  for  every  subject.  We 
suggest  that  calibration  computations  employ  a  polynomial  curve  fit 
technique  similar  to  that  used  by  current  ASL  eye  tracking  systems  (ref, 
37),  which  will  probably  require  about  17  target  points  spread  over  tne 
visual  field  of  interest.  Interpolation  techniques  would  require 
significantly  more  target  points  to  achieve  comparable  accuracy  over  the 
large  VPO  range  and  would  result  In  too  lengthy  a  callbraton  procedure. 

The  suggested  protocol  for  pilot  self  calibration  1$  as  follows: 

1.  Pilot  requests  calibration  procedure  with  a  manual  switch  or 
voice  command. 

2.  The  VPO  displays  one  target  point. 

3.  The  pilot  fixates  the  target  point  and  activates  a  manual 
switch  while  fixating  the  target. 

4.  If  the  eye  tracker  did  not  have  proper  eye  recognition  when 
the  data  entry  switch  was  activated,  an  appropriate  message  Is 
displayed  so  the  pilot  will  know  to  try  again  or  take 
corrective  action. 

5.  If  the  eye  tracker  did  have  proper  recognition,  then  the 
next  target  point  Is  displayed  and  step  3  Is  repeated. 

6.  After  the  last  target  point  is  entered  and  calibration 
computations  completed  by  the  eye  tracker,  poi nt-of-gaze 
feedback  (SVFB)  is  displayed  along  with  all  target  points  so 
that  the  pilot  can  check  performance. 
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7.  Another  switch  activation  by  the  pilot  will  end  the 
calibration  procedure. 

6.6  Processor 


The  processor  design  objectives  are  to  make  as  much  image 
Information  as  possible  available  for  detection,  recognition,  and  center 
computation  algorithms,  while  providing  enough  computational  power  to 
achieve  desired  temporal  resolution.  The  processing  system  architecture 
should  be  appropriate  for  240  Hz  operation,  as  well  as  for  a  60  Hz 
prototype. 


Considering  the  first  objective.  It  Is  desirable  to  have  entire  grey 
scale  Images  available  In  an  Image  memory.  It  may,  however,  take  too 
much  time  to  access  all  of  these  pixels  from  Image  memory  In  order  to 
find  the  Image  features  of  Interest.  We  will,  therefore,  plan  to  use  a 
preprocessor  to  perform  threshold  edge  detection  and  histogram 
computation  tasks  “on  the  fly"  as  the  video  data  comes  from  the  camera. 

The  main  processor  can  quickly  determine  probable  locations  of  Important 
features  from  the  edge  Information  and,  when  necessary,  sections  of  the 
raw  Image  memory  can  be  accessed  for  more  detailed  local  processing. 

A  possible  high  speed  (up  to  240  Hz)  design  based  on  the  VHE  bus  and 
Motorola  68020  processor  Is  dlagranmed  In  figure  6.11.  The  camera 
controller  as  discussed  In  section  6.3  will  be  responsible  for  synchro¬ 
nizing  to  an  external  signal  from  the  host,  providing  drive  signals  to 
the  camera,  and  passing  video  data  to  the  A/D  converter.  A  60  Hz 
prototype  version  of  the  eye  tracker  will  differ  from  figure  6.11  only 
In  that  a  more  conventional  60  Hz  CCD  camera  will  be  used  In  place  of 
the  Reticon  H9128,  and  the  custom  camera  controller  can  be  replaced  with 
a  standard  video  sync  signal  generator  and  genlock  circuit. 

The  front  end  processor  (or  preprocessor)  will  perform  computationally 
Intensive  operations  on  the  A/D  output  data  "on  the  fly"  and  store  the 
results  In  memory.  Specifically,  the  processor  will  store  the  entire 
Image  In  memory  and  will  also  store  addresses  of  threshold  crossing 
points,  the  address  of  the  brightest  pixel  or  pixels  In  the  Image,  and 
Intensity  histogram  data.  The  edge  Information  Is  comparable  to  the 
Image  data  available  to  current  eye  tracker  processors.  The  raw  Image 
memory  will  significantly  enhance  the  capabilities  for  feature  detection 
and  recognition.  Histogram  Information  can  be  used  by  the  main 
processor  to  determine  threshold  values  for  subsequent  fields. 


The  main  processor,  shown  In  figure  6.11  as  a  Motorola  68020,  will 
be  responsible  for  most  feature  recognition,  eye  position  computation 
and  communication  with  the  host  VPD  system.  Benchmark  comparisons  with 
the  processor  currently  used  for  ASL  eye  trackers  have  shown  a  7.5  times 
speed  advantage  fur  the  63020,  and  this  may  be  sufficient  for  the  higher 
speed  and  Increased  computations  needed. 


If  a  single  68020  processor  is  not  sufficient,  a  special  purpose 
high  speed  processor,  as  shown  In  figure  6.11,  can  be  Included  and  used 
for  computationally- Intensive  portions  of  the  algorithm.  Such  a 
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Figure  6.11  Processing  system  for  VPD  eye  tracker. 


processor,  for  example  a  40  MIPS,  bit  slice  level  device,  will 
definitely  provide  the  necessary  computational  power,  but  should  be  used 
very  sparingly  because  of  the  high  overhead  associated  with  the 
microcode  programming  that  these  devices  require.  To  the  extent  that 
algorithm  functions  can  be  implemented  in  parallel,  processing  power  can 
also  be  enhanced  by  adding  an  additional  68020  module  to  the  VME  bus. 

The  hardware  used  should  be  the  minimum  required  for  the  computational 
task  and  these  specific  decisions  can  be  made  most  effectively  after  the 
algorithm  is  successfully  tested  off-line. 

Data  will  be  transferred  to  the  VPD  host  system  through  a  parallel 
port  compatible  with  the  appropriate  Q22  or  Unibus  interface.  Each  data 
field  will  be  composed  of  5  words,  as  listed  below: 

1.  Eye  azimuth  position 

2.  Eye  elevation  position 

3.  Pupil  diameter 

4.  Status  (error  conditions,  etc.) 

5.  Check  sum  (sum  of  previous  4  words  to  check  for  transmission 
errors ) 

The  eye  tracker  will  also  be  interfaced  to  the  VPD  host  through  a 
standard  RS232  serial  port  from  which  the  host  will  be  able  to  initiate 
and  control  the  calibration  sequence,  request  offset  corrections,  etc. 

It  is  very  important,  especially  during  initial  phases  of  simulator 
research,  that  sufficient  feedback  be  available  to  qualitatively 
evaluate  eye  tracker  performance  in  real-time  and  to  understand  the 
nature  of  any  problems  encountered.  For  this  reason,  two  displays  are 
shown  in  figure  6.11.  One  display  will  be  the  video  eye  image  captured 
by  the  detector  with  indicators  superimposed  to  show  system  recognition 
and  center  computation.  A  second  display  will  show  the  resulting  line- 
of-gaze  computation  as  a  cursor  or  cross  hair  moving  over  the  visual 
fiel  d. 

The  eye  tracker  transport  delay  will  probably  be  three  fields 
(update  intervals)  as  shown  by  the  timing  diagram  in  figure  6.12.  If 
the  main  processor  completes  computations  in  less  than  one  field,  the 
delay  may  be  slightly  shorter,  but  will  always  be  greater  than  two 
fields. 

Nothing  in  the  processing  system  proposed  is  beyond  current  state- 
of-the-art,  and  most  components  are  commercially  available.  The  camera 
controller  for  high  speed  (240  Hz)  operation  and  the  front  end  processor 
will  be  custom  components.  Some  of  the  non-VME  bus  interfaces  between 
components  may  also  be  custom  designed.  Motorola  68000  series  CPU  and 
floating  point  processor  combinations  have  been  developed  for  programs 
requiring  in-flight  use  with  other  VPD  systems  and  other  AAMRL 
developments,  and  thus  meet  appropriate  mil -spec  requirements  for 
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flight.  When  detailed  specifications  become  available,  these  processors 
should  be  evaluated  and  considered  for  a  VPD  eye  tracker. 

The  VME  bus  and  68020  processor  were  chosen  for  consistency  with 
other  VPD  components  and  because  VME  is  a  reasonably  fast  bus  that  has 
become  something  of  an  industrial  standard.  The  same  design  framework 
would  be  possible  using  an  Intel  80386  processor  and  an  ISM  bus. 

Because  the  IBM  bus  is  relatively  slow,  the  image  memory  would  probably 
have  to  be  Interfaced  directly  to  the  high  speed  port  available  on  the 
80386  processor,  and  an  IBM  bus  design  would  not  be  consistent  with 
other  VPD  components.  There  may,  however,  be  some  advantage  to  IBM  In 
terms  of  component  cost,  variety  of  commercially  available  components, 
and  variety  of  available  software  development  tools. 

The  algorithm  can  be  developed  and  tested  off-line  using  digitally 
stored  images.  An  off-line  development  system  must  Include  a  processor, 
a  hard  disk,  a  video  frame  grabber,  a  display  terminal  and  suitable 
software  development  tools.  One  of  the  standard  80286  or  80386  based 
systems  now  available,  plus  an  added  frame  grabber,  would  be  sufficient 
for  this  task.  This  class  of  system  offers  the  advantages  of  relatively 
low  cost,  a  large  quantity  and  variety  of  compatible  frame  grabber 
boards,  and  most  importantly,  an  enormous  number  of  excellent  software 
development  tools  which  will  significantly  facilitate  the  algorithm 
development  task. 

6.7  Performance  Goals 

The  following  specifications  represent  reasonable  performance  goals 
for  the  VPD  eye  tracker.  Although  not  exactly  the  same  as  the  ideal 
specifications  listed  in  5.7,  the  most  important  aspects  are  preserved 
and  the  goals  are  realistic. 


Inner 

Tracking  Range: 

+20  deg  vertical  and 

horizontal 

Outer 

Tracking  Range: 

field  between  inner  and  total  range 

Accuracy: 

0.7  deg  within  inner 
2.5  deg  within  outer 

range 

range 

Precision: 

0.35  deg 

Resolution: 

0.25  deg 

Update  Rate: 

60  Hz  --  first  prototype 

240  Hz  --  enhanced  prototype 

Transport 

Delay: 

3  update  intervals 

Pupil  Diameter 
Range: 

2-8  mm 
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Total  Eye 

Tracking  Range:  +40  deg  horizontal;  +30  deg  (up)  -25  deg 

Tdown)  vertical 

Optics  Adjustment;  requires  1  minute  or  less  after  VPO 

helmet  donned 


Calibration 

Procedure: 


Tolerance  for 
Helmet  Slippage: 


-  requires  1  min  or  less 

-  can  be  self  administered  by  pilot 

-  quick  offset  adjustment  capability 
after  calibration 


4  mm  at  the  eye 


Perhaps  the  most  important  performance  goal  can  best  be  stated 
qualitatively:  the  system  must  perform  dependably  with  the  nonideal 
images  and  artifacts  that  will  often  be  present  in  a  complex  environment 
like  the  VPO.  These  "real  world"  considerations  are  the  driving  force 
behind  the  design  approaches  outlined  in  the  preceding  sections  and  the 
development  program  outlined  In  the  following  section. 


7.0  VPO  EYE  TRACKER  OEVELOPMENT  PROGRAM 

A  logical  sequence  of  tasks  for  a  VPO  eye  tracker  development 
program  is  presented  below. 

1.  Obtain  detailed  dimensions  and  optical  specifications  for 
the  VPO  design  to  be  used. 

2.  Build  bright  and  dark  pupil  breadboard  optics  packages  to 
test  with  a  prototype  VPO  helmet  using  one  of  the  optical  path 
approaches  discussed  in  section  6.2.  The  breadboard  package 
should  include  an  illuminator,  standard  60  Hz  solid  state 
video  camera,  and  necessary  relay  optics.  Components  need  not 
be  packaged  nor  need  they  be  rigidly  fastened  to  the  helmet, 
but  rather  the  helmet  can  be  stabilized  in  some  way  (e.g., 
subject  chin  rest)  and  components  held  by  lab  stands  cr 
temporarily  fastened  to  the  helmet. 

3.  Using  real  subjects,  experiment  with  the  breadboard  optics 
to  obtain  the  most  suitable  camera  eye  image.  These  tests 
should  be  done  both  with  the  VPO  display  projector  off  and  with 
the  display  on.  When  the  display  is  on,  tests  should  be  done 
with  the  full  range  of  scene  luminance  levels  likely  to  be 
experienced  in  actual  use.  If  the  prototype  system  is  to  be 
used  in  flight,  then  daylight  conditions  should  be  tested  as 
Wcl  1  . 

4.  Video  tape  the  resulting  eye  camera  images  and,  if  possible, 
use  a  computer  and  video  frame  grabber  to  digitally  record  some 
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of  the  Images.  It  will  be  important  to  collect  images  at  a  wide 
range  of  eye  positions  and  images  exhibiting  samples  of  all 
expected  artifacts. 

5.  If  necessary,  revise  the  optics  breadboards  and  repeat  step 
3  and  4. 

6.  In  parallel  with  steps  1  through  5,  set  up  a  development 
system  for  off-line  work  on  the  eye  tracker  algorithms.  As 
discussed  in  section  6.5,  this  system  must  include  a  processor, 
a  hard  disk,  a  video  frame  grabber,  a  display  terminal,  and 
suitable  software  development  tools. 

7.  Digitize  selected  images  from  video  tape  data  collected  in 
steps  3  through  5  and  use  the  data  to  develop  and  test  pupil  and 
CR  recognition  and  center  computation  algorithms. 

From  an  algorithmic  point  of  view,  the  difference  between 
the  60  Hz  image  and  future  higher  sample  rate  images  will  be 
lower  spatial  resolution  of  the  high  speed  images  (i.e., 
detectors  with  fewer  pixels).  Lower  spatial  resolution  can 
easily  be  simulated  by  mathematically  combining  neighboring 
pixel  values.  Algorithms  should  be  tested  with  simulated  128  x 
128  pixel  resolution,  as  well  as  with  full  resolution. 

8.  Based  on  the  results  of  step  7  and  the  framework  presented 
in  section  6.5,  decide  whether  to  use  bright  or  dark  pupil 
optics  and  specify  the  real  time  processing  system  for  a 
prototype  VPO  eye  tracker.  Although  it  is  suggested  that  the 
first  prototype  be  a  60  Hz  system,  the  processing  system  should 
be  suitable  for  up  to  240  Hz  operation. 

9.  Implement  the  processor  design  specified  in  step  8. 

10.  In  parallel  with  step  9,  specify  the  final  optics  module 
design  for  the  60  Hz  version  prototype  system  and  construct  a 
"brassboard"  version. 

11.  Test  the  optics  module  brassboard  with  the  VPD  system, 
revise  if  necessary,  and  construct  the  final  optics  module  for 
the  60  Hz  prototype  eye  tracker. 

12.  Test  the  completed  processor  system. 

13.  Test  the  combined  optics  and  processor  system  in  real  time. 

14.  Deliver  the  60  Hz  prototype  VPO  eye  tracker  and  test  with 
the  prototype  VPO  system. 

15.  Develop  high  speed  camera  controller,  high  speed  detector 
repackaging,  and  optics  module  modifications  for  high  speed 
enhancement  to  VPD  eye  tracker. 
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16.  Test  breadboard  versions  of  high  speed  components. 

17.  Deliver  and  Install  enhancements  for  240  Hz  eye  tracker 

operation.  - 
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